### Training Setup Configuration

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This output details the configuration for the training run, including the number of examples, epochs, batch sizes, and optimization steps. It also notes ignored columns that do not correspond to the model's forward arguments.

```text
Output:
The following columns in the training set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: lang, text.
***** Running training *****
  Num examples = 4802
  Num Epochs = 1
  Instantaneous batch size per device = 32
  Total train batch size (w. parallel, distributed & accumulation) = 32
  Gradient Accumulation steps = 1
  Total optimization steps = 151
```

--------------------------------

### Training Examples Progress

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This progress bar shows the current step in processing training examples, with 4802 total examples.

```text
Result:
  0%|          | 0/4802 [00:00<?, ?ex/s]
```

--------------------------------

### Install pysentimiento

Source: https://github.com/pysentimiento/robertuito/blob/main/README.md

Install the pysentimiento library to use RoBERTuito. This is a prerequisite for preprocessing tweets.

```bash
pip install pysentimiento
```

--------------------------------

### Training Setup and Configuration

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

Illustrates the output during the setup of a PyTorch training process, including device configuration and training arguments. It highlights changes in default arguments for future versions.

```text
Output:
PyTorch: setting up devices
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
The following columns in the training set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: lang, text.
***** Running training *****
  Num examples = 4802
  Num Epochs = 1
  Instantaneous batch size per device = 32
  Total train batch size (w. parallel, distributed & accumulation) = 32
  Gradient Accumulation steps = 1
  Total optimization steps = 151
```

--------------------------------

### Load TensorBoard Extension and Start Server

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Pretrain.ipynb

Loads the tensorboard extension for Jupyter notebooks and starts the tensorboard server, directing logs to the './logs' directory.

```python
%load_ext tensorboard
%tensorboard --logdir ./logs

```

--------------------------------

### Install Development Dependencies

Source: https://github.com/pysentimiento/robertuito/blob/main/README.md

Install poetry and project dependencies for development. This is required for contributing to the RoBERTuito project.

```bash
pip install poetry
poetry install
```

--------------------------------

### Evaluation Examples Progress

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This progress bar indicates the current step in processing evaluation examples, with 2443 total examples.

```text
Result:
  0%|          | 0/2443 [00:00<?, ?ex/s]
```

--------------------------------

### Install and Upgrade Libraries

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Finetuning_BERT_on_tweets.ipynb

Installs or upgrades numpy and installs the transformers and datasets libraries from Hugging Face. Ensure compatibility with existing torch versions.

```bash
!pip install -U numpy
!pip install git+https://github.com/huggingface/transformers/ datasets
```

--------------------------------

### Evaluation Setup and Saving

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This output details the configuration for the evaluation run, noting ignored columns. It also shows the process of saving the model checkpoint and its associated configuration and weights.

```text
Output:
The following columns in the evaluation set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: lang, text.
***** Running Evaluation *****
  Num examples = 2443
  Batch size = 32
Saving model checkpoint to /tmp/tmpffo2c7rw/checkpoint-151
Configuration saved in /tmp/tmpffo2c7rw/checkpoint-151/config.json
Model weights saved in /tmp/tmpffo2c7rw/checkpoint-151/pytorch_model.bin


Training completed. Do not forget to share your model on huggingface.co/models =)


Loading best model from /tmp/tmpffo2c7rw/checkpoint-151 (score: 0.639641523361206).
The following columns in the evaluation set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: lang, text.
***** Running Evaluation *****
  Num examples = 7264
  Batch size = 32
```

--------------------------------

### Total Evaluation Examples Progress

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This progress bar shows the current step in processing a larger set of evaluation examples, with 7264 total examples.

```text
Result:
  0%|          | 0/7264 [00:00<?, ?ex/s]
```

--------------------------------

### Install and Update Libraries

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Finetuning_BERT_on_tweets.ipynb

Installs and updates essential libraries for natural language processing, including transformers, datasets, and their dependencies. Ensure you have the latest versions for optimal performance and compatibility.

```bash
pip install -U "transformers==4.9.0.dev0" datasets sacremoses
```

--------------------------------

### Training Configuration Output

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Finetuning_BERT_on_tweets.ipynb

Displays the training configuration details, including the number of training steps and batch sizes. This output confirms the training setup before execution.

```text
Output:
max_steps is given, it will override any value given in num_train_epochs
***** Running training *****
  Num examples = 20480
  Num Epochs = 50
  Instantaneous batch size per device = 64
  Total train batch size (w. parallel, distributed & accumulation) = 2048
  Gradient Accumulation steps = 4
  Total optimization steps = 500

```

--------------------------------

### Training Configuration Summary

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/misc/Irony.ipynb

Provides a summary of the training configuration, including the number of examples, epochs, batch sizes, and optimization steps. Note that 'text' and 'lang' columns are ignored during training.

```text
The following columns in the training set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: text, lang.
***** Running training *****
  Num examples = 5400
  Num Epochs = 5
  Instantaneous batch size per device = 32
  Total train batch size (w. parallel, distributed & accumulation) = 32
  Gradient Accumulation steps = 1
  Total optimization steps = 845
```

--------------------------------

### Training Progress Bar Example

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

Shows a typical progress bar output during data processing or training steps, indicating the percentage of completion and processing speed.

```text
Result:
  0%|          | 0/151 [00:00<?, ?ba/s]
```

```text
Result:
  0%|          | 0/77 [00:00<?, ?ba/s]
```

```text
Result:
  0%|          | 0/227 [00:00<?, ?ba/s]
```

--------------------------------

### Train MLM with DeepSpeed on GPU

Source: https://github.com/pysentimiento/robertuito/blob/main/TRAINING.md

Command to train a Masked Language Model (MLM) using DeepSpeed on GPUs. This setup requires DeepSpeed to be installed and configured. It specifies model, data paths, and training parameters like batch size, accumulation steps, and number of steps.

```bash
model="models/twerto-base-uncased"
num_proc=24 #Check your CPU cores
num_steps=10
batch_size=32
acc=64
output_dir="models/twerto-base-uncased-${num_steps}"
deepspeed --num_gpus 2 bin/run_mlm.py\
    --input_dir data/filtered_tweets/ --output_dir $output_dir --model_name $model \
    --num_steps $num_steps  --per_device_batch_size $batch_size --accumulation_steps $acc\
    --eval_steps 500 --save_steps 2000
    --num_proc $num_proc --on_the_fly
```

--------------------------------

### Run MLM with run_mlm_hf.py

Source: https://github.com/pysentimiento/robertuito/blob/main/TRAINING.md

Example command to run the MLM training script using `run_mlm_hf.py`. This script likely integrates with Hugging Face libraries. Note the presence of both `eval_steps` and `save_steps` defined twice; ensure the intended values are used.

```bash
python bin/xla_spawn.py --num_cores 8 bin/run_mlm_hf.py\
    --adam_beta1 0.9 --adam_beta2 0.98 --adam_epsilon 0.000001\
    --eval_steps 1000 --save_steps 3000 --logging_steps 50\
    --per_device_batch_size
    --input_dir data/filtered_tweets/ --output_dir $output_dir --model_name $model \
    --num_steps $num_steps  --per_device_batch_size $pdbs --accumulation_steps $acc\
    --learning_rate $lr --on_the_fly
    --eval_steps 500 --save_steps 2000
    --num_proc $num_proc
```

--------------------------------

### Dataset Loading and Training Output

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Finetuning_BERT_on_tweets.ipynb

Shows the output messages during dataset loading and the start of the training process. This indicates the progress and readiness of the model for training.

```text
Output:
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!
Loading datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'special_tokens_mask', 'token_type_ids'],
    num_rows: 20480
})
Done
Training!

```

--------------------------------

### Train MLM from Scratch on v2 TPU

Source: https://github.com/pysentimiento/robertuito/blob/main/TRAINING.md

Command for training a Masked Language Model (MLM) from scratch on a v2 TPU. This configuration uses a higher number of steps and different batch/accumulation settings compared to the initial scratch training example. Ensure `max_eval_steps` is appropriate.

```bash
export TPU_IP_ADDRESS=10.97.22.154
export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470"

model="models/twerto-base-uncased"
num_proc=16 #Check your CPU cores
num_steps=200000
pdbs=64
acc=16
lr=0.0006
eval_steps=2500
save_steps=5000
logging_steps=500
warmup_ratio=0.06
tok_batch_size=16384
output_dir="models/twerto-base-uncased-${num_steps}"
python bin/xla_spawn.py --num_cores 8 bin/run_mlm.py\
    --input_dir data/filtered_tweets/ --output_dir $output_dir --model_name $model \
    --num_steps $num_steps  --per_device_batch_size $pdbs --accumulation_steps $acc \
    --learning_rate $lr --warmup_ratio $warmup_ratio \
    --tok_batch_size $tok_batch_size \
    --eval_steps $eval_steps --save_steps $save_steps --logging_steps $logging_steps --max_eval_steps 100 \
    --num_proc $num_proc
```

--------------------------------

### Load NLP Datasets and Tokenizers

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Vocabulary Comparison.ipynb

Loads various NLP datasets (hate, sentiment, emotion, irony, context_hate) and tokenizers for different models. This setup is essential for preparing data and models for comparison.

```python
%load_ext autoreload
%autoreload 2
import importlib

from finetune_vs_scratch.hate import load_datasets as load_hate
from finetune_vs_scratch.sentiment import load_datasets as load_sentiment
from finetune_vs_scratch.emotion import load_datasets as load_emotion
from finetune_vs_scratch.irony import load_datasets as load_irony
from finetune_vs_scratch.context_hate import load_datasets as load_context_hate
from finetune_vs_scratch.model import load_tokenizer

tasks = {
    "context_hate": load_context_hate,
    "hate": load_hate,
    "sentiment": load_sentiment,
    "emotion": load_emotion,
    "irony": load_irony,
}

tokenizers = {
    "beto_uncased" : load_tokenizer("dccuchile/bert-base-spanish-wwm-uncased", 128),
    "beto_cased" : load_tokenizer("dccuchile/bert-base-spanish-wwm-cased", 128),
    "roberta" : load_tokenizer("BSC-TeMU/roberta-base-bne", 128),
    "bertin" : load_tokenizer("bertin-project/bertin-roberta-base-spanish", 128),
    "robertuito_cased" : load_tokenizer("finiteautomata/robertuito-base-cased", 128),
    "robertuito_uncased" : load_tokenizer("finiteautomata/robertuito-base-uncased", 128),
    "robertuito_deacc" : load_tokenizer("finiteautomata/robertuito-base-deacc", 128),
}
```

--------------------------------

### Load RoBERTa Base Tokenizer

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization with wrapping.ipynb

Loads the tokenizer for the 'roberta-base' model from the transformers library. This is a common setup step for using pre-trained models.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
```

--------------------------------

### Model Training and Evaluation Output

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This output shows the standard logs during model training and evaluation, including the number of examples, batch size, and saving checkpoints. It also indicates ignored columns during evaluation.

```text
The following columns in the evaluation set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: lang, text.
***** Running Evaluation *****
  Num examples = 2443
  Batch size = 32
Saving model checkpoint to /tmp/tmp80o8hioa/checkpoint-151
Configuration saved in /tmp/tmp80o8hioa/checkpoint-151/config.json
Model weights saved in /tmp/tmp80o8hioa/checkpoint-151/pytorch_model.bin


Training completed. Do not forget to share your model on huggingface.co/models =)


Loading best model from /tmp/tmp80o8hioa/checkpoint-151 (score: 0.6329007744789124).
The following columns in the evaluation set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: lang, text.
***** Running Evaluation *****
  Num examples = 7264
  Batch size = 32
```

--------------------------------

### Get Second Batch of Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/iterators.ipynb

Retrieves and lists the second batch of data from the dataset iterator. Demonstrates sequential access to batches.

```python
list(next(batches))
```

--------------------------------

### Get Current Working Directory

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/test.ipynb

Prints the current working directory. This command is often used for debugging or verifying the execution environment.

```shell
!pwd
```

--------------------------------

### Calculate Token Lengths

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Tweets Exploratory.ipynb

Iterates through the tokenized dataset to calculate the number of actual tokens (excluding special start/end tokens) for each example.

```python
from tqdm.auto import tqdm
lens = []
for i in tqdm(range(len(dataset))):
    lens.append(sum(dataset[i]["attention_mask"]) - 2)

```

--------------------------------

### Get First Batch of Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/iterators.ipynb

Retrieves and lists the first batch of data from a dataset iterator. Shows the raw string content of the tweets in the batch.

```python
batches = dataset.batches

list(next(batches))
```

--------------------------------

### Tokenize Text and Calculate Lengths

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Vocabulary Comparison.ipynb

Tokenizes the text data using different tokenizers and calculates the number of tokens for each instance. Subtracts 2 to account for special start and end tokens.

```python
# Les resto 2 por los tokens iniciales y finales

args = {
    "truncation": True,
    "max_length": 128,
}

for tok_name, tokenizer in tokenizers.items():
    df[tok_name] = [sum(x) - 2 for x in tokenizer(list(df["text"].values), **args)["attention_mask"]]
```

--------------------------------

### Get Second Encoded Batch with Special Tokens

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/iterators.ipynb

Retrieves the second batch of encoded data and decodes it, including special tokens. Shows how special tokens are represented in the decoded output.

```python
encodings = next(encoded_batches)

for enc in encodings:
    print(tokenizer.decode(enc["input_ids"])).
```

--------------------------------

### Run Sentiment Analysis Experiments with Collator

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This function orchestrates the sentiment analysis pipeline, including model and tokenizer loading, dataset preparation, tokenization with padding, and model training using `Trainer`. It handles data collator setup and metric computation.

```python
from transformers import DataCollatorWithPadding

def run_with_collator(model_name, device, data_path=None, limit=None, epochs=5, batch_size=32, eval_batch_size=32, padding=False, **kwargs):
    """
    Run sentiment analysis experiments
    """
    print("Running sentiment experiments")

    model, tokenizer = load_model_and_tokenizer(model_name, num_labels=len(label2id), device=device)
    train_dataset, dev_dataset, test_dataset = load_datasets(data_path=data_path, limit=limit)

    def tokenize(batch):
        return tokenizer(batch['text'], padding=padding, truncation=True)

    accumulation_steps = 32 // batch_size

    train_dataset = train_dataset.map(tokenize, batched=True, batch_size=batch_size)
    dev_dataset = dev_dataset.map(tokenize, batched=True, batch_size=eval_batch_size)
    test_dataset = test_dataset.map(tokenize, batched=True, batch_size=eval_batch_size)


    output_path = tempfile.mkdtemp()
    training_args = TrainingArguments(
        output_dir=output_path,
        num_train_epochs=epochs,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=eval_batch_size,
        gradient_accumulation_steps=accumulation_steps,
        warmup_ratio=0.1,
        evaluation_strategy="epoch",
        do_eval=False,
        weight_decay=0.01,
        logging_dir='./logs',
        load_best_model_at_end=True,
        metric_for_best_model="macro_f1",
        **kwargs,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=DataCollatorWithPadding(tokenizer, padding="longest"),
        compute_metrics=lambda x: compute_sentiment_metrics(x, id2label=id2label),
        train_dataset=train_dataset,
        eval_dataset=dev_dataset,
    )

    trainer.train()

    test_results = trainer.evaluate(test_dataset)

    os.system(f"rm -Rf {output_path}")
    return test_results

```

--------------------------------

### Initialize and Use Custom Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Initializes the custom dataset with file paths and iterates through it to process lines. Uses `glob` to find files and `tqdm` for progress visualization.

```python
from glob import glob

tweet_files = glob("../data/filtered_tweets/*.txt")[:1]
my_ds = MyCustomDataset(
    files=tweet_files
)
```

```python
from tqdm.auto import tqdm
for x in tqdm(my_ds):
    pass
```

--------------------------------

### Run Benchmarks

Source: https://github.com/pysentimiento/robertuito/blob/main/README.md

Execute benchmarks for RoBERTuito models. Specify the model name, number of times to run, and an output path for the results.

```bash
python bin/run_benchmark.py <model_name> --times 5 --output_path <output_path>
```

--------------------------------

### Get an Encoding from the Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Retrieves a single encoding from the `BatchProcessedDataset` iterator to inspect its structure and content.

```python
encoding = (next(iter(my_ds)))

encoding
```

--------------------------------

### Load and Prepare Text Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Pretrain.ipynb

Loads a text dataset from local files, splits it into training and testing sets, and prints the dataset structure. It also demonstrates how to set a transformation for on-the-fly tokenization if needed.

```python
import random 
from glob import glob 
from datasets import load_dataset
import os

input_dir = "../data/filtered_tweets"
num_files = 5

tweet_files = random.sample(
    glob(os.path.join(input_dir, "*.txt")),
    num_files
)

print(f"Selecting {tweet_files}")

train_files, test_files = tweet_files[:-1], tweet_files[-1:]

dataset = load_dataset("text", data_files={"train": train_files, "test": test_files})


train_dataset, test_dataset = dataset["train"], dataset["test"]

```

--------------------------------

### Initialize and Use Hugging Face Tokenizers

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization naive.ipynb

Initializes RobertaTokenizerFast with custom vocab and merge files, and loads the pre-trained vinai/bertweet-base tokenizer. It then tokenizes a sample sentence using the RobertaTokenizerFast.

```python
from transformers import RobertaTokenizerFast, AutoTokenizer

roberta_tokenizer = RobertaTokenizerFast(vocab_file, merges_file, never_split=special_tokens)
bertweet_tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base")

(roberta_tokenizer("@usuario tugo bierno skere comunista")["input_ids"])
```

--------------------------------

### Get First Encoded Batch

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/iterators.ipynb

Retrieves the first batch of encoded data from the dataset. Iterates through the encoded items and decodes them, skipping special tokens.

```python
encoded_batches = dataset.encoded_batches

encodings = next(encoded_batches)

for enc in encodings:
    print(tokenizer.decode(enc["input_ids"], skip_special_tokens=True))
```

--------------------------------

### Loading Pre-trained Model and Tokenizer

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

Demonstrates loading a pre-trained Spanish BERT model and its tokenizer from Hugging Face. This involves downloading weights and configuration files if not already cached.

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "dccuchile/bert-base-spanish-wwm-uncased"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
```

--------------------------------

### Train MLM from Scratch

Source: https://github.com/pysentimiento/robertuito/blob/main/TRAINING.md

Shell command to initiate training of a Masked Language Model (MLM) from scratch on a TPU. This involves setting up TPU environment variables and specifying model parameters, input/output directories, and training configurations.

```bash
export TPU_IP_ADDRESS=10.110.227.66
export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470"

model="models/twerto-base-uncased"
num_proc=16 #Check your CPU cores
num_steps=100000
pdbs=128
acc=8
lr=0.0006
eval_steps=3000
save_steps=6000
warmup_ratio=0.06
tok_batch_size=16384
output_dir="models/twerto-base-uncased-${num_steps}"
python bin/xla_spawn.py --num_cores 8 bin/run_mlm.py\
    --input_dir data/filtered_tweets/ --output_dir $output_dir --model_name $model \
    --num_steps $num_steps  --per_device_batch_size $pdbs --accumulation_steps $acc \
    --learning_rate $lr --warmup_ratio $warmup_ratio \
    --tok_batch_size $tok_batch_size \
    --eval_steps $eval_steps --save_steps $save_steps --logging_steps 100 --max_eval_steps 100 \
    --num_proc $num_proc
```

--------------------------------

### Encode and Get Tokens

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization naive.ipynb

Encodes a sample tweet string using the trained SentencePieceBPETokenizer and retrieves the resulting tokens. This demonstrates how the tokenizer breaks down text into subword units.

```python
vocab = tokenizer.get_vocab()
inv_vocab = {v:k for k, v in vocab.items()}


tokenizer.encode("Qué hacesssss @usuario").tokens
```

--------------------------------

### Run Benchmark for Beto Cased

Source: https://github.com/pysentimiento/robertuito/blob/main/docs/RUN_BENCHMARKS.md

Execute benchmark tests for the 'dccuchile/bert-base-spanish-wwm-cased' model. This command benchmarks the cased version of the Beto model, specifying the run count, output file, and the irony task.

```bash
python bin/run_benchmark.py 'dccuchile/bert-base-spanish-wwm-cased' --times 5 output/beto-cased.json --task irony
```

--------------------------------

### Load Datasets and Run Irony Detection

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/misc/Irony.ipynb

This snippet loads the necessary datasets and initiates the irony detection experiments using the Robertuito model on a CUDA-enabled device. Ensure the 'finetune_vs_scratch.irony' module is available.

```python
%load_ext autoreload
%autoreload 2
from finetune_vs_scratch.irony import load_datasets, run

run("finiteautomata/robertuito-base-uncased", "cuda")
```

--------------------------------

### Initialize and Iterate Naive Processed Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Initializes the `NaiveProcessedDataset` and iterates through it, showing progress with `tqdm`. This demonstrates line-by-line tokenization.

```python
my_ds = NaiveProcessedDataset(
    files=tweet_files
)

for x in tqdm(my_ds):
    pass
```

--------------------------------

### Initialize and Configure SentencePieceBPETokenizer

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization with wrapping.ipynb

Initializes a SentencePieceBPETokenizer and configures its normalizers and post-processor. This includes adding special tokens and defining rules for text cleaning and normalization.

```python
from tokenizers import SentencePieceBPETokenizer, BertWordPieceTokenizer, ByteLevelBPETokenizer
from tokenizers import normalizers, Regex
from tokenizers.processors import RobertaProcessing
from finetune_vs_scratch.preprocessing import special_tokens
from finetune_vs_scratch.tokenizer import tokenizer_special_tokens

tokenizer = SentencePieceBPETokenizer()
#replacement="_"

tokenizer.add_special_tokens(tokenizer_special_tokens)

strip_accents = True
lowercase = True

tokenizer_normalizers = [
    normalizers.NFKC(),
    normalizers.BertNormalizer(
        clean_text=True,
        handle_chinese_chars=True,
        strip_accents=strip_accents,
        lowercase=lowercase,
    ),
    normalizers.Replace(Regex("(\\W)?@usuario(\\W)"), " @usuario "),
    normalizers.Replace("hashtag", " hashtag "),
    # Error de preprocesamiento
    normalizers.Replace(Regex("(\\W)url(\\W)"), " url "),
    normalizers.Replace("http://url", " url "),
]

tokenizer.normalizer = normalizers.Sequence(tokenizer_normalizers)

vocab = tokenizer.get_vocab()

tokenizer.post_processor = RobertaProcessing(
    cls=("<s>", tokenizer.token_to_id("<s>")),
    sep=("</s>", tokenizer.token_to_id("</s>")),
)
```

--------------------------------

### Load Tokenized Datasets from Disk

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/test.ipynb

Loads pre-tokenized training and testing datasets from disk using the datasets library.

```python
import datasets
from datasets import load_dataset, Features, Value, load_from_disk

train_dataset = load_from_disk("../data/arrow/tokenized_train")
test_dataset = load_from_disk("../data/arrow/tokenized_test")

train_dataset, test_dataset
```

--------------------------------

### Configure Training Arguments

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Pretrain.ipynb

Defines a dictionary of arguments for the training process, including evaluation steps, saving steps, logging steps, batch sizes, gradient accumulation, learning rate, weight decay, and warmup ratio.

```python
eval_steps = 10
save_steps = 50


args = {
    "eval_steps":eval_steps,
    "save_steps":save_steps,
    "logging_steps": 50,
    "per_device_train_batch_size": 32,
    "per_device_eval_batch_size": 32,
    "gradient_accumulation_steps": 2,
    "learning_rate": 5e-4,
    "weight_decay": 0.1,
    "warmup_ratio": 0.06,
}

```

--------------------------------

### Normalize String with Tokenizer

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization with wrapping.ipynb

Demonstrates string normalization using the configured tokenizer's normalizer.

```python
tokenizer.normalizer.normalize_str("@usuariotugo")
```

```python
tokenizer.normalizer.normalize_str("..url..")
```

--------------------------------

### Run Benchmark for Beto Uncased

Source: https://github.com/pysentimiento/robertuito/blob/main/docs/RUN_BENCHMARKS.md

Execute benchmark tests for the 'dccuchile/bert-base-spanish-wwm-uncased' model. Specify the number of times to run the benchmark and the output file. This is useful for evaluating the performance of the uncased Beto model on the irony task.

```bash
python bin/run_benchmark.py 'dccuchile/bert-base-spanish-wwm-uncased' --times 5 output/beto-uncased.json --task irony
```

--------------------------------

### Run Benchmark for Robertuito Uncased

Source: https://github.com/pysentimiento/robertuito/blob/main/docs/RUN_BENCHMARKS.md

Execute benchmark tests for the 'finiteautomata/robertuito-base-uncased' model. This snippet benchmarks the Robertuito uncased model, specifying 5 runs, an output file, and the irony task.

```bash
python bin/run_benchmark.py 'finiteautomata/robertuito-base-uncased' --times 5 output/robertuito-uncased-200k.json --task irony
```

--------------------------------

### Finetune MLM on v2 TPU

Source: https://github.com/pysentimiento/robertuito/blob/main/TRAINING.md

Command to finetune a Masked Language Model (MLM) on a v2 TPU. Requires setting TPU IP address and XRT configuration. Adjust batch size, accumulation steps, and other parameters as needed.

```bash
export TPU_IP_ADDRESS=10.97.22.154
export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470"
pdbs=64
acc=4
lr=0.0006
num_proc=16
num_steps=5000
warmup_ratio=0.07
output_dir="models/beto-uncased-${num_steps}/"
python bin/xla_spawn.py --num_cores 8 bin/run_mlm.py\
    --input_dir data/filtered_tweets/ --output_dir $output_dir --num_steps $num_steps \
    --model_name 'dccuchile/bert-base-spanish-wwm-uncased' \
    --per_device_batch_size $pdbs --accumulation_steps 4 \
    --warmup_ratio $warmup_ratio \
    --num_proc $num_proc --finetune
```

--------------------------------

### Iterate Through Hugging Face Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Iterates through the loaded Hugging Face dataset using `tqdm` for progress tracking. Demonstrates processing of the 'train' split.

```python
from tqdm.auto import tqdm
for x in tqdm(dataset["train"]):
    pass
```

--------------------------------

### Model Initialization Warning

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/misc/Irony.ipynb

This output indicates that some model weights were not used during initialization, which is expected when using a pre-trained model for a different task. New weights for the classifier will be initialized.

```text
Some weights of the model checkpoint at dccuchile/bert-base-spanish-wwm-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at dccuchile/bert-base-spanish-wwm-cased and are newly initialized: ['classifier.weight', 'classifier.bias', 'bert.pooler.dense.weight', 'bert.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

--------------------------------

### Load and Prepare Tweet Files

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization with wrapping.ipynb

Loads and prepares tweet data for tokenization training. It sets up autoreload and selects a subset of tweet files.

```python
%load_ext autoreload
%autoreload 2
from glob import glob

num_files = 100
tweet_files = glob("../../data/filtered_tweets/*.txt")


train_files = tweet_files[:2]


tweets = list([x.strip("\n") for x in open(tweet_files[0])])[:1_00_000]
```

--------------------------------

### Finetune MLM on v3 TPU

Source: https://github.com/pysentimiento/robertuito/blob/main/TRAINING.md

Command to finetune a Masked Language Model (MLM) on a v3 TPU. This configuration uses a larger number of processors and fewer steps compared to v2. Ensure 'on_the_fly' is suitable for your dataset.

```bash
num_proc=32
num_steps=25000; python bin/xla_spawn.py --num_cores 8 bin/run_mlm.py --input_dir data/filtered_tweets/ --output_dir models/beto-uncased-${num_steps}/ --num_steps $num_steps --model_name 'dccuchile/bert-base-spanish-wwm-uncased' --per_device_batch_size 128 --accumulation_steps 2 --num_proc $num_proc  --on_the_fly --finetune
```

--------------------------------

### Initialize and Iterate Batch Processed Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Initializes the `BatchProcessedDataset` with a specified batch size and iterates through it using `tqdm` for progress visualization. This demonstrates batch tokenization.

```python
my_ds = BatchProcessedDataset(
    files=tweet_files,
    batch_size=1024,
)
for x in tqdm(my_ds):
    pass
```

--------------------------------

### Iterate Through Hugging Face Dataset for Comparison

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Iterates through the Hugging Face dataset's 'train' split using `tqdm` to establish a baseline for performance comparison with custom datasets.

```python
for ex in tqdm(dataset["train"]):
    pass
```

--------------------------------

### Model Initialization Warnings

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

These warnings indicate that some model weights were not used during initialization, which is expected when initializing from a checkpoint of a different task or architecture. New weights are initialized for the sequence classification head.

```text
Some weights of the model checkpoint at dccuchile/bert-base-spanish-wwm-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at dccuchile/bert-base-spanish-wwm-uncased and are newly initialized: ['bert.pooler.dense.weight', 'classifier.weight', 'bert.pooler.dense.bias', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

--------------------------------

### Initialize and Configure BERT Tokenizer

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/test.ipynb

Initializes a fast BERT tokenizer from a pre-trained model and adds custom special tokens. Sets the maximum sequence length for the tokenizer.

```python
import torch
from transformers import AutoTokenizer
from finetune_vs_scratch.preprocessing import special_tokens
from transformers import BertForMaskedLM, BertTokenizerFast
model_name = 'dccuchile/bert-base-spanish-wwm-uncased'

tokenizer = BertTokenizerFast.from_pretrained(model_name)
tokenizer.model_max_length = 128
tokenizer.add_tokens(special_tokens)
```

--------------------------------

### Load Autoreload and Prepare Tweet Files

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization naive.ipynb

Loads the autoreload extension and prepares training files by globbing text files from a specified directory. It selects a subset of files and loads a specified number of tweets from the first file.

```python
%load_ext autoreload
%autoreload 2
from glob import glob

num_files = 100
tweet_files = glob("../../data/filtered_tweets/*.txt")

train_files = tweet_files[:2]


tweets = list([x.strip("\n") for x in open(tweet_files[0])])[:100_000]
```

--------------------------------

### Load Datasets and Run Irony Detection

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/misc/Irony.ipynb

Loads datasets and initiates the irony detection experiment using a specified pre-trained model and device. Ensure 'autoreload' is enabled for development.

```python
%load_ext autoreload
%autoreload 2
from finetune_vs_scratch.irony import load_datasets, run

run("dccuchile/bert-base-spanish-wwm-cased", "cuda")
```

--------------------------------

### Initialize Batch Processed Dataset

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/iterators.ipynb

Initializes a BatchProcessedDataset with specified training files, tokenizer, batch size, and limit. This prepares the dataset for iteration.

```python
import glob

train_files = glob.glob("../data/tweets/train/*")


dataset = ds.BatchProcessedDataset(
    train_files, tokenizer, batch_size=1024, limit=1_000_000
)
```

--------------------------------

### Load Dataset with Hugging Face Datasets

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Loads a text dataset using the Hugging Face `datasets` library. This is an alternative to custom dataset loading for standard text file formats.

```python
from datasets import load_dataset

dataset = load_dataset("text", data_files=tweet_files)
```

--------------------------------

### Execution Time Measurement

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

This output provides the CPU time and wall time taken for the execution of a process, offering insights into performance.

```text
Output:
CPU times: user 1min 25s, sys: 30.2 s, total: 1min 55s
Wall time: 1min 50s
```

--------------------------------

### Initialize SentencePieceBPETokenizer

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization naive.ipynb

Initializes a SentencePieceBPETokenizer. This tokenizer is suitable for subword tokenization using the Byte Pair Encoding algorithm.

```python
from tokenizers import SentencePieceUnigramTokenizer, SentencePieceBPETokenizer, BertWordPieceTokenizer, ByteLevelBPETokenizer

tokenizer = SentencePieceBPETokenizer()#replacement="_"
```

--------------------------------

### Initialize Data Collator and DataLoader

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Pretrain.ipynb

Sets up a DataCollatorForLanguageModeling for masked language modeling and creates a DataLoader for the training dataset.

```python
from transformers import DataCollatorForLanguageModeling
from torch.utils.data.dataloader import DataLoader

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

dataloader = DataLoader(
    train_dataset,
    batch_size=2048,
    collate_fn=data_collator,
)

```

--------------------------------

### Load and Test Multiple Tokenizers

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization with wrapping.ipynb

Loads multiple tokenizers ('deacc', 'uncased', 'cased') from specified model paths and performs a sanity check by encoding and decoding sample text. Shows how different casing and accent handling affects tokenization.

```python
from transformers import AutoTokenizer

tokenizers = {
    "deacc": "../../models/twerto-base-deacc-uncased",
    "uncased": "../../models/twerto-base-uncased",
    "cased": "../../models/twerto-base-cased",
}

tokenizers = {k: AutoTokenizer.from_pretrained(v) for k, v in tokenizers.items()}

for model_name, tokenizer in tokenizers.items():
    print("="*80)
    print(model_name, "\n"*3)
    print("Sanity check")
    print(f"@usuario => {tokenizer.encode('@usuario')}")
    text = ["esta es una PRUEBA EN MAYÚSCULAS Y CON TILDES @usuario @usuario", "ATR cumbia gato hashtag"]
    print(f"{text}\n{tokenizer.decode(tokenizer.encode(*text))}")
```

--------------------------------

### Save and Load Hugging Face Tokenizer

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization with wrapping.ipynb

Saves the configured Hugging Face tokenizer to a local directory and then loads it back using AutoTokenizer for verification.

```python
transformer_tokenizer.save_pretrained("small")
```

```python
from transformers import AutoTokenizer
transformer_tokenizer = AutoTokenizer.from_pretrained("small")
```

--------------------------------

### Load and Preprocess Sentiment Datasets

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/improve_tokenization.ipynb

Loads sentiment analysis datasets from a CSV file, preprocesses the text, and splits the data into training, development, and testing sets. It defines the features including text, language, and a ClassLabel for polarity.

```python
"""
Run sentiment experiments
"""
import os
import pathlib
import tempfile
import pandas as pd
from finetune_vs_scratch.model import load_model_and_tokenizer
from finetune_vs_scratch.preprocessing import preprocess
from transformers import Trainer, TrainingArguments
from datasets import Dataset, Value, ClassLabel, Features
from pysentimiento.tass import id2label, label2id
from pysentimiento.metrics import compute_metrics as compute_sentiment_metrics


project_dir = pathlib.Path("..\n")
data_dir = os.path.join(project_dir, "data")
sentiment_dir = os.path.join(data_dir, "sentiment")


def load_datasets(data_path=None, limit=None):
    """
    Load sentiment datasets
    """
    features = Features({
        'text': Value('string'),
        'lang': Value('string'),
        'label': ClassLabel(num_classes=3, names=["neg", "neu", "pos"])
    })
    data_path = data_path or os.path.join(sentiment_dir, "tass.csv")
    df = pd.read_csv(data_path)
    df["label"] = df["polarity"].apply(lambda x: label2id[x])
    df["text"] = df["text"].apply(lambda x: preprocess(x))

    train_dataset = Dataset.from_pandas(df[df["split"] == "train"], features=features)
    dev_dataset = Dataset.from_pandas(df[df["split"] == "dev"], features=features)
    test_dataset = Dataset.from_pandas(df[df["split"] == "test"], features=features)


    if limit:
        """
        Smoke test
        """
        print("\n\n", f"Limiting to {limit} instances")
        train_dataset = train_dataset.select(range(min(limit, len(train_dataset))))
        dev_dataset = dev_dataset.select(range(min(limit, len(dev_dataset))))
        test_dataset = test_dataset.select(range(min(limit, len(test_dataset))))


    return train_dataset, dev_dataset, test_dataset

```

--------------------------------

### Full Training Function with XLA Support

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/Pretrain.ipynb

A comprehensive function to train a model, handling dataset loading, tokenization (optional on-the-fly), data collation, and setting up the Hugging Face Trainer with provided training arguments. It includes a note on using MpModelWrapper for XLA memory management.

```python
from transformers import Trainer, TrainingArguments
from transformers import DataCollatorForLanguageModeling

import os
import torch
import torch_xla.core.xla_model as xm
import torch_xla.distributed.parallel_loader as pl
import torch_xla.distributed.xla_multiprocessing as xmp
from datasets import load_from_disk
from datasets import load_dataset

import sys


"""
Todo esto es para evitar que la memoria salte por los aires por usar 'fork' en XLA
Ver: https://pytorch.org/xla/release/1.8/index.html#torch_xla.distributed.xla_multiprocessing.MpModelWrapper

"""

WRAPPED_MODEL = xmp.MpModelWrapper(model)
tokenize_on_the_fly = False

def train_model(
    model, tokenizer, **kwargs,
    ):
    """
    """ 

    print("Loading datasets")
    train_dataset = load_from_disk(train_path)
    test_dataset = load_from_disk(test_path)
   
    train_dataset = train_dataset.remove_columns(["text"])
    test_dataset = test_dataset.remove_columns(["text"])
    print(train_dataset)
    if tokenize_on_the_fly:
        train_dataset.set_transform(tokenize)
        test_dataset.set_transform(tokenize)
    print("Done")

    
    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=True, mlm_probability=0.15
    )

    model_path = f"./{model_name}"


    training_args = TrainingArguments(
        **kwargs
    )

    print("Training!")
    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
    )

    trainer.train()

```

--------------------------------

### Iterate Through Dataset with TQDM

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/iterators.ipynb

Iterates through the entire dataset using tqdm for progress visualization. This is a common pattern for processing large datasets.

```python
from tqdm.auto import tqdm

for x in tqdm(dataset):
    pass
```

--------------------------------

### Prepare Data for Analysis

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/deacc-vs-uncased.ipynb

Organizes the loaded model performance data into a pandas DataFrame. It iterates through defined tasks and models, extracting relevant metrics.

```python
import pandas as pd
tasks = {
    "context_hate": "eval_mean_f1",
    "hate": "eval_macro_f1",
    "sentiment": "eval_macro_f1",
    "emotion": "eval_macro_f1",
    "irony": "eval_macro_f1",
}

data = []
models = ["robertuito-cased", "robertuito-uncased", "robertuito-deacc"]

for task, metric in tasks.items():
    for model in models:
        for run in outs[model][task]:
            data.append({
                "model": model,
                "task": task,
                "macro_f1": run[metric],
            })
df = pd.DataFrame(data)
```

--------------------------------

### Create Dataset from Dictionary

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/test.ipynb

Creates a Hugging Face Dataset object from a Python dictionary containing text data.

```python
from datasets import Dataset

dataset = Dataset.from_dict({"text": tweets})
```

--------------------------------

### Load Tokenizer from Pre-trained Model

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/datasets.ipynb

Loads a pre-trained tokenizer from the Hugging Face Hub. Ensure the model path is correct.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("../models/twerto-base-uncased")
```

--------------------------------

### Convert to Hugging Face PreTrainedTokenizerFast

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/tokenization/Tokenization with wrapping.ipynb

Converts the custom SentencePieceBPETokenizer into a Hugging Face PreTrainedTokenizerFast object, mapping special tokens for compatibility.

```python
from transformers import PreTrainedTokenizerFast

transformer_tokenizer = PreTrainedTokenizerFast(
    tokenizer_object=tokenizer,
    bos_token="<s>",
    eos_token="</s>",
    sep_token="</s>",
    cls_token="<s>",
    unk_token="<unk>",
    pad_token="<pad>",
    mask_token="<mask>",
)
```

--------------------------------

### Load Model Outputs

Source: https://github.com/pysentimiento/robertuito/blob/main/notebooks/deacc-vs-uncased.ipynb

Loads JSON output files from a specified directory for different models. It filters out 'test' files and stores the results in a dictionary keyed by model name.

```python
%load_ext autoreload
%autoreload 2
import os
import glob
import json

outs = {}


for filename in sorted([f for f in glob.glob("../output/*.json") if "test" not in f]): 
    model_name = os.path.basename(filename).split(".")[0]
    print(model_name)
    with open(filename) as f:
        outs[model_name] = json.load(f)
```