### Install Project Dependencies

Source: https://github.com/ucsc-real/tokencleaning/blob/main/README.md

Installs the required Python packages for running training, evaluation, or inference. Ensure PyTorch is installed beforehand.

```bash
pip install -r requirements.txt
```

--------------------------------

### Tokenize Prompt and Completion Data

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Tokenizes training examples with separate prompt and completion fields. Masks prompt tokens from loss and handles spacing and BOS tokens automatically. Requires `transformers` and `functools`.

```python
from transformers import AutoTokenizer
from functools import partial
from scripts.generate_token_label import encode_with_prompt_completion_format

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B")

example = {
    "prompt": "Translate to French: Hello world",
    "completion": "Bonjour le monde"
}

result = encode_with_prompt_completion_format(
    example,
    tokenizer=tokenizer,
    max_seq_length=2048,
    with_prompt_token=False,  # mask prompt tokens from loss (standard SFT)
    add_bos=False,
)
# result["input_ids"]      -- full token sequence (prompt + completion + EOS)
# result["labels"]         -- -100 for prompt positions, token IDs for completion
# result["attention_mask"] -- all ones (no padding applied at this stage)
print(result["labels"])  # tensor([-100, -100, ..., 1045, 5765, 8312, 2])
```

--------------------------------

### End-to-end Fixed-Model Cleaning Pipeline

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Orchestrates the Fixed-Model Cleaning workflow, including computing per-token losses, generating labels, and fine-tuning the base model. Requires prerequisite setup for the reference model.

```bash
# Prerequisites: edit cluster_root_path and ensure warmup reference model exists
# (run get_ref_model.sh first to train the 10k warmup reference model)
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

# Step 1: Train the warmup reference model on 10k samples (one-time setup)
bash get_ref_model.sh

# Step 2: Run Fixed-Model Cleaning pipeline
bash fixed_model_cleaning.sh
# Internally runs:
#   bash_src/calculate_loss.sh  <base_model>  <train_data> 6 8
#   bash_src/calculate_loss.sh  <ref_model>   <train_data> 6 8
#   python scripts/generate_token_label.py --select_token_level global --data_prop 0.6 ...
#   bash_src/finetune.sh        <base_model>  <train_data> 6 8 <model_path> 0.6 token_cleaning 42
```

--------------------------------

### Token Selection Algorithms

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Provides functions for selecting top-k influential tokens globally across the dataset or per sample. Global selection enforces a dataset-wide budget, while sample-level selection preserves a proportional budget per example.

```python
from scripts.generate_token_label import get_global_top_k_indices, get_sample_top_k_indices
import torch

# raw_labels: list of label lists (response tokens have real IDs; prompt tokens are -100)
# all_losses: list of loss-difference lists (loss_base - loss_ref) per token
raw_labels = [[-100, -100, 101, 202, 303], [-100, 404, 505]]
all_losses  = [[0.0,  0.0, 1.2, 0.3, 0.8], [0.0, 0.9, 0.1]]
```

--------------------------------

### Run MMLU Evaluation with Accelerate

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

This command launches a multi-GPU evaluation for the MMLU benchmark using Accelerate. Ensure the model path and output path are correctly configured.

```bash
accelerate launch --multi-gpu --main_process_port 29519 --num_processes 8     -m lm_eval --model hf     --model_args "pretrained=/checkpoints/Llama-3.2-3B/lora_merged_ds2-50k-fixed-model,dtype=bfloat16"     --tasks mmlu     --batch_size 16     --num_fewshot 5     --limit 0.99     --output_path eval_results/Llama-3.2-3B/0.6/ds2-50k-fixed-model     --seed 42
```

--------------------------------

### Fine-tune Llama-3.2-3B with Token Cleaning

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Launches a fine-tuning job for Llama-3.2-3B using token cleaning labels. Requires specific configurations for mixed precision, LoRA, and data proportion.

```bash
accelerate launch \
    --num_machines 1 \
    --mixed_precision bf16 \
    --num_processes 8 \
    --config_file fsdp_configs/fsdp_config.yaml \
    --main_process_port 29501 \
    scripts/finetune.py \
    --model_name_or_path meta-llama/Llama-3.2-3B \
    --tokenizer_name meta-llama/Llama-3.2-3B \
    --train_file raw_data/ds2-50k-fixed-model.json \
    --max_seq_length 2048 \
    --use_lora \
    --lora_rank 64 \
    --lora_alpha 16 \
    --lora_dropout 0.1 \
    --gradient_checkpointing \
    --per_device_train_batch_size 6 \
    --gradient_accumulation_steps 1 \
    --learning_rate 1e-4 \
    --lr_scheduler_type linear \
    --warmup_ratio 0.03 \
    --num_train_epochs 1 \
    --output_dir /checkpoints/Llama-3.2-3B/lora_ds2-50k-fixed-model/ \
    --token_select_pattern token_cleaning \
    --data_prop 0.6 \
    --with_prompt_token False \
    --seed 42
```

--------------------------------

### Multi-task Evaluation with lm-eval-harness

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Evaluates a fine-tuned model across multiple benchmarks using lm-eval-harness. Results are saved to per-task JSON files.

```bash
# Edit run_eval.sh to point to your model and data paths, then:
bash run_eval.sh
```

--------------------------------

### Run Model Evaluation

Source: https://github.com/ucsc-real/tokencleaning/blob/main/README.md

Execute the evaluation script for the model. Ensure lm-eval-harness is set up and TydiQA data is prepared.

```bash
bash run_eval.sh
```

--------------------------------

### Run TydiQA Evaluation with Python Script

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

This command executes the TydiQA evaluation script. Configure the data directory, number of shots, and model path as needed.

```bash
python -m eval.tydiqa.run_eval     --data_dir eval_data/eval/tydiqa/     --n_shot 1     --max_num_examples_per_lang 200     --max_context_length 512     --save_dir eval_results/Llama-3.2-3B/0.6/ds2-50k-fixed-model     --model_name_or_path /checkpoints/Llama-3.2-3B/lora_merged_ds2-50k-fixed-model     --eval_batch_size 5
```

--------------------------------

### FSDP Distributed Training Configuration

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

This YAML configuration file sets up PyTorch Fully Sharded Data Parallel (FSDP) for multi-GPU training. It enables features like transformer-based auto-wrapping and activation checkpointing for efficient training.

```yaml
# fsdp_configs/fsdp_config.yaml
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
mixed_precision: bf16
num_machines: 1
num_processes: 8                          # match to CUDA_VISIBLE_DEVICES count
fsdp_config:
  fsdp_sharding_strategy: FULL_SHARD      # shard params + grads + optimizer states
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_activation_checkpointing: true     # save GPU memory at cost of recompute
  fsdp_cpu_ram_efficient_loading: true    # load weights without duplicating in RAM
  fsdp_offload_params: false              # keep params on GPU for speed
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: false
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
  fsdp_state_dict_type: SHARDED_STATE_DICT
```

--------------------------------

### Generate Token-Level Training Labels (Sample)

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Computes token influence scores by differencing base and reference model losses. Retains top-k influential tokens per sample. Output labels mask non-influential tokens with -100.

```bash
python scripts/generate_token_label.py \
    --base_model_name_or_path meta-llama/Llama-3.2-3B \
    --ref_model_name_or_path /checkpoints/Llama-3.2-3B/lora_merged_ds2-10k-warmup \
    --tokenizer_name_or_path meta-llama/Llama-3.2-3B \
    --train_data raw_data/ds2-50k-fixed-model.json \
    --data_prop 0.6 \
    --select_token_level sample
```

--------------------------------

### Split Dataset into Ordered Subsets

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Divides a training dataset into equal-sized, non-overlapping sequential chunks and saves each as a JSON file. Used in iterative training pipelines to process data in portions.

```bash
# Split DS2-50k into 5 ordered subsets of ~10k samples each
python scripts/generate_subset.py \
    --train_dataset_name ds2-50k-self-evolving \
    --subset_size 5 \
    --data_path raw_data \
    --org_data_name jlpang888/DS2_50k
# Creates: raw_data/ds2-50k-self-evolving_0.json  (~10k samples)
#          raw_data/ds2-50k-self-evolving_1.json
#          raw_data/ds2-50k-self-evolving_2.json
#          raw_data/ds2-50k-self-evolving_3.json
#          raw_data/ds2-50k-self-evolving_4.json
```

--------------------------------

### Project Citation

Source: https://github.com/ucsc-real/tokencleaning/blob/main/README.md

Use this BibTeX entry when citing the Token Cleaning project in academic work.

```bibtex
@article{pang2025token,
  title={Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning},
  author={Pang, Jinlong and Di, Na and Zhu, Zhaowei and Wei, Jiaheng and Cheng, Hao and Qian, Chen and Liu, Yang},
  journal={arXiv preprint arXiv:2502.01968},
  year={2025}
}
```

--------------------------------

### End-to-end Self-Evolving Cleaning Pipeline

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Orchestrates the iterative Self-Evolving Cleaning workflow, where each iteration uses the previous one's merged model as a reference. This script loops over data subsets for training.

```bash
# Self-Evolving Cleaning: iterates over 5 data subsets, each ~10k samples
# Subset 0: warmup fine-tune (no token cleaning, builds first reference model)
# Subsets 1-4: token cleaning with the previous iteration's model as reference
bash self_evolving_cleaning.sh

# Equivalent manual steps for iteration idx=2:
REF_MODEL=/checkpoints/Llama-3.2-3B/data_prop_0.6/lora_merged_ds2-50k-self-evolving_1
TRAIN_DATA=raw_data/ds2-50k-self-evolving_2.json

bash bash_src/calculate_loss.sh meta-llama/Llama-3.2-3B $TRAIN_DATA 6 8
bash bash_src/calculate_loss.sh $REF_MODEL              $TRAIN_DATA 6 8

python scripts/generate_token_label.py \
    --base_model_name_or_path meta-llama/Llama-3.2-3B \
    --ref_model_name_or_path  $REF_MODEL \
    --train_data $TRAIN_DATA \
    --data_prop 0.6 \
    --select_token_level global

bash bash_src/finetune.sh $REF_MODEL $TRAIN_DATA 6 8 \
    /checkpoints/Llama-3.2-3B/data_prop_0.6 0.6 token_cleaning 42
```

--------------------------------

### Run Fixed-Model Cleaning Pipeline

Source: https://github.com/ucsc-real/tokencleaning/blob/main/README.md

Executes the fixed-model cleaning process. This involves first obtaining a reference model and then applying the cleaning script.

```bash
bash get_ref_model.sh
bash fixed_model_cleaning.sh
```

--------------------------------

### Generate Token-Level Training Labels (Global)

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Computes token influence scores by differencing base and reference model losses. Retains top-k influential tokens globally. Output labels mask non-influential tokens with -100.

```bash
python scripts/generate_token_label.py \
    --base_model_name_or_path meta-llama/Llama-3.2-3B \
    --ref_model_name_or_path /checkpoints/Llama-3.2-3B/lora_merged_ds2-10k-warmup \
    --tokenizer_name_or_path meta-llama/Llama-3.2-3B \
    --train_data raw_data/ds2-50k-fixed-model.json \
    --data_prop 0.6 \
    --select_token_level global
```

--------------------------------

### Evaluate and Print Model Results

Source: https://github.com/ucsc-real/tokencleaning/blob/main/read_results.ipynb

This script iterates through specified models and tasks, collects evaluation results from JSON files, aggregates them into a pandas DataFrame, and prints a formatted table. It handles different metric types for various tasks and calculates an average score.

```python
import os
from lm_eval.utils import make_table
import json
import pandas as pd

eval_root_path="eval_results"
TASK_LISTS=['mmlu', 'bbh', 'gsm8k', 'truthfulqa_mc2', "arc_challenge", "piqa", "hellaswag", "openbookqa", "triviaqa", 'sciq', 'arc_easy', 'logiqa', 'boolq', 'winogrande'] ##task


base_model="meta-llama/Llama-3.2-3B" #"meta-llama/Llama-3.1-8B" "mistralai/Mistral-7B-v0.3"
data_prop = 0.6

##eval model name
model_tags=["ds2-50k-self-evolving"]


results_all = {}
for model_tag in model_tags:
    eval_result_path = f"{eval_root_path}/{os.path.basename(base_model)}/{data_prop}/{model_tag}/"

    if model_tag != 'base':
        exp_files = os.listdir(eval_result_path)
        for file_name in exp_files:
            if str(data_prop) in file_name and os.path.basename(base_model) in file_name: 
                log_path = file_name
    else:
        log_path = os.listdir(eval_result_path)[0] 
 
    json_files = os.listdir(os.path.join(eval_result_path, log_path))

    results = {}
    for file in json_files:
        with open(os.path.join(eval_result_path, log_path, file), 'r') as f:
            temp = json.load(f)
            for task in TASK_LISTS:
                if task in temp['results'].keys():                    
                    if task in ['hellaswag', 'piqa', 'openbookqa', 'arc_challenge', 'mmlu', 'truthfulqa_mc2', 'sciq', 'arc_easy', 'logiqa', 'boolq', 'winogrande']:
                        metric = 'acc,none'
                    elif task == 'gsm8k':
                        metric = 'exact_match,strict-match'
                    elif task == "triviaqa":
                        metric = "exact_match,remove_whitespace"
                    elif task == 'bbh':
                        metric = 'exact_match,get-answer' 
                    results[task] = temp['results'][task][metric]

    ## load tydiqa result 
    tydiqa_result_file = os.path.join(eval_result_path, "metrics.json")
    if os.path.exists(tydiqa_result_file):
        with open(tydiqa_result_file, 'r') as f:
            results['tydiqa'] = round(json.load(f)['average']['f1'] / 100, 4)

    results_all[model_tag] = results


results_df = pd.DataFrame.from_dict(results_all, orient='index')
TASK_LISTS=["truthfulqa_mc2", "tydiqa", 'logiqa', 'mmlu',  "hellaswag", "arc_challenge", "boolq"]

results_df = results_df[TASK_LISTS]
results_df = results_df.map(lambda x: round(100*x, 2) if pd.notnull(x) else x)
results_df['Average'] = results_df.mean(axis=1).round(1)

print("\nResults DataFrame (Reordered with Average, Percentage Format):\n")
results_df = results_df.reindex(model_tags)
print(results_df.to_string(line_width=1000))
```

--------------------------------

### Merge LoRA Weights into Base Model

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Merges trained LoRA adapter weights into the base model, handling vocabulary embedding resizing and optional qLoRA dequantization. The output model is ready for inference.

```bash
# Merge LoRA adapter into the base Llama-3.2-3B model
python scripts/merge_lora.py \
    --base_model_name_or_path meta-llama/Llama-3.2-3B \
    --lora_model_name_or_path /checkpoints/Llama-3.2-3B/lora_ds2-50k-fixed-model/ \
    --output_dir /checkpoints/Llama-3.2-3B/lora_merged_ds2-50k-fixed-model/ \
    --save_tokenizer \
    --use_fast_tokenizer
```

--------------------------------

### Run Self-Evolving Cleaning Pipeline

Source: https://github.com/ucsc-real/tokencleaning/blob/main/README.md

Executes the self-evolving cleaning process, which follows an iterative approach for token cleaning.

```bash
bash self_evolving_cleaning.sh
```

--------------------------------

### Select Top Tokens for Label Cleaning

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Selects top tokens based on global loss across the dataset or independently per sample. Used to build cleaned label tensors by unmasking selected tokens.

```python
global_indices = get_global_top_k_indices(raw_labels, all_losses, data_prop=0.6)
# Returns: [(0, 2), (1, 1), (0, 4)]  -- (sample_idx, token_idx) pairs

sample_indices = get_sample_top_k_indices(raw_labels, all_losses, data_prop=0.6)
# Returns: [(0, 2), (0, 4), (1, 1)]  -- preserves proportions within each sample

# Build the cleaned label tensor
selected_labels = [[-100] * len(l) for l in raw_labels]
for i, j in global_indices:
    selected_labels[i][j] = raw_labels[i][j]  # unmask selected tokens
torch.save(selected_labels, "results/label/token_labels_ds2-50k.pt")
```

--------------------------------

### Calculate Per-Token Cross-Entropy Loss

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Computes per-token cross-entropy losses using a base or reference model. Supports multi-GPU inference via FSDP. The output is used for downstream influence score computation.

```bash
accelerate launch \
    --num_processes 8 \
    --config_file fsdp_configs/fsdp_config.yaml \
    --main_process_port 29501 \
    scripts/calculate_token_loss.py \
    --model_name_or_path meta-llama/Llama-3.2-3B \
    --tokenizer_name meta-llama/Llama-3.2-3B \
    --train_file raw_data/ds2-50k-fixed-model.json \
    --max_seq_length 2048 \
    --per_device_train_batch_size 6 \
    --num_train_epochs 1 \
    --reduce_loss sum \
    --with_prompt_token False
```

--------------------------------

### Tokenize Multi-Turn Chat Data

Source: https://context7.com/ucsc-real/tokencleaning/llms.txt

Tokenizes multi-turn conversations from a list of messages. Masks non-assistant turns from training loss and supports max-length truncation. Raises `ValueError` for unknown roles or empty lists. Requires `transformers` and `functools`.

```python
from transformers import AutoTokenizer
from scripts.generate_token_label import encode_with_messages_format
from functools import partial

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B")

example = {
    "messages": [
        {"role": "system",    "content": "You are a helpful assistant."},
        {"role": "user",      "content": "What is 2+2?"},
        {"role": "assistant", "content": "The answer is 4."},
    ]
}

result = encode_with_messages_format(
    example,
    tokenizer=tokenizer,
    max_seq_length=2048,
    with_prompt_token=False,  # only train on assistant responses
    add_bos=False,
)
# result["labels"] masks system + user turns with -100,
# only assistant response tokens carry real IDs
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.