### Dockerfile for NeMo Toolkit

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Sets up a Docker container for fine-tuning tasks using the NeMo toolkit. It installs the NeMo toolkit with NLP support from a specific branch.

```dockerfile
# Dockerfile-nemo: Container for fine-tuning tasks
FROM nvcr.io/nvidia/pytorch:21.02-py3

# Install NeMo toolkit with NLP support
BRANCH=nlp_mp_ddp
RUN pip install git+https://github.com/NVIDIA/NeMo.git@${BRANCH}#egg=nemo_toolkit[all]

# Build and run:
# docker build -f Dockerfile-nemo -t gatortron-nemo .
# docker run --gpus all -v /data:/data gatortron-nemo python ner_expr.py --ner_config config.yaml
```

--------------------------------

### Dockerfile for Megatron-LM Pretraining

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Configures a Docker container for pretraining tasks using Megatron-LM. It clones the Megatron-LM repository and installs its dependencies.

```dockerfile
# Dockerfile-megatron: Container for pretraining
FROM nvcr.io/nvidia/pytorch:21.02-py3

# Clone and setup Megatron-LM
RUN git clone https://github.com/NVIDIA/Megatron-LM.git && \
    cd Megatron-LM && \
    pip install -r requirements.txt

# Build and run:
# docker build -f Dockerfile-megatron -t gatortron-megatron .
# docker run --gpus all -v /data:/data gatortron-megatron python pretrain_bert.py ...
```

--------------------------------

### Load GatorTron Model and Tokenizer with Hugging Face

Source: https://github.com/uf-hobi-informatics-lab/gatortron/blob/main/README.md

Use this snippet to load the GatorTron-medium model and its corresponding tokenizer from Hugging Face. Ensure you have the 'transformers' and 'huggingface_hub' libraries installed. The model is suitable for natural language understanding tasks.

```python
from huggingface_hub import login
from transformers import AutoModel, AutoTokenizer, AutoConfig

tokinizer= AutoTokenizer.from_pretrained('UFNLP/gatortron-medium')
config=AutoConfig.from_pretrained('UFNLP/gatortron-medium')
mymodel=AutoModel.from_pretrained('UFNLP/gatortron-medium')

encoded_input=tokinizer("Bone scan:  Negative for distant metastasis.", return_tensors="pt")
encoded_output = mymodel(**encoded_input)

# then you can feed encoded_output to downstream task layers for different usecases e.g., NER, RE, MRC etc.
```

--------------------------------

### Build Singularity image from definition file

Source: https://github.com/uf-hobi-informatics-lab/gatortron/blob/main/docker/readme.md

Requires root or sudo privileges to build a container from a local definition file.

```bash
sudo singularity build nemo.sif nemo.def
```

--------------------------------

### Launch Distributed Pretraining with SLURM

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Configures and executes distributed pretraining for GatorTron-medium using SLURM and torch.distributed.launch.

```bash
#!/bin/bash
#SBATCH --nodes=124
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-task=8
#SBATCH --cpus-per-task=128
#SBATCH --mem=2000gb
#SBATCH --time=120:00:00
#SBATCH --partition=hpg-ai

# Configuration for GatorTron-medium (3.9B parameters)
VOCAB_FILE=./vocabs/vocab.txt
CHECKPOINT_PATH=./checkpoints/gatortron_medium
DATA_PATH=./data/bin/clinical_notes_NOTE_TEXT_sentence

# Model architecture (3.9B: L=48, H=2560, A=40)
BERT_ARGS="--num-layers 48 \
    --hidden-size 2560 \
    --num-attention-heads 40 \
    --seq-length 512 \
    --max-position-embeddings 512 \
    --lr 0.0001 \
    --lr-decay-iters 500000 \
    --train-iters 1000000 \
    --min-lr 0.00001 \
    --lr-warmup-fraction 0.01 \
    --micro-batch-size 8 \
    --global-batch-size 3968 \
    --vocab-file $VOCAB_FILE \
    --split 949,50,1 \
    --tensor-model-parallel-size 2 \
    --pipeline-model-parallel-size 1 \
    --tokenizer-type BertWordPieceCase \
    --fp16"

OUTPUT_ARGS="--log-interval 1000 \
    --save-interval 10000 \
    --eval-interval 5000 \
    --eval-iters 100 \
    --checkpoint-activations"

# Launch distributed training
python -m torch.distributed.launch \
    --nproc_per_node=$SLURM_GPUS_PER_TASK \
    --nnodes=$SLURM_JOB_NUM_NODES \
    --node_rank=$SLURM_NODEID \
    --master_addr=$PRIMARY \
    --master_port=$PRIMARY_PORT \
    ./Megatron-LM/pretrain_bert.py \
    $BERT_ARGS $OUTPUT_ARGS \
    --save $CHECKPOINT_PATH \
    --load $CHECKPOINT_PATH \
    --data-path $DATA_PATH
```

--------------------------------

### Singularity Environment for HPC

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Provides instructions for building and running a Singularity container, an alternative to Docker for HPC environments. It demonstrates building from a Docker image and executing a Python script with GPU support.

```bash
# Singularity alternative for HPC environments
# Build from Docker
singularity build nemo.sif docker://nvcr.io/nvidia/pytorch:21.02-py3

# Run with GPU support
singularity exec --nv nemo.sif python ner_expr.py --ner_config config.yaml --gpus 8
```

--------------------------------

### Build Singularity images from Docker Hub

Source: https://github.com/uf-hobi-informatics-lab/gatortron/blob/main/docker/readme.md

Create .sif files directly from existing Docker images hosted on NVIDIA's container registry.

```bash
singularity build megatron.sif docker://nvcr.io/nvidia/pytorch:21.02-py3
```

```bash
singularity build nemo.sif docker://nvcr.io/nvidia/nemo:1.0.0b3
```

--------------------------------

### Question Answering Configuration and Training

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

YAML configuration for QA tasks and the corresponding Python script to initialize, train, and run inference with a QAModel.

```yaml
trainer:
  gpus: 8
  max_epochs: 5
  precision: 16
  accelerator: ddp

model:
  dataset:
    data_dir: ./data/qa/emrqa
    train_fn: medication-train.json
    dev_fn: medication-test.json
    test_fn: medication-test.json
    max_seq_length: 512
    max_query_length: 96
    max_answer_length: 96
    doc_stride: 384

  tokenizer:
    tokenizer_name: megatron-bert-cased
    vocab_file: ./vocabs/vocab.txt

  language_model:
    pretrained_model_name: megatron-bert-cased
    lm_checkpoint: ./checkpoints/gatortron_9b/iter_0670000
    config_file: ./checkpoints/gatortron_9b/config.json

  optim:
    name: adamw
    lr: 1e-5
```

```python
from nemo.collections import nlp as nemo_nlp
from nemo.utils.exp_manager import exp_manager
import pytorch_lightning as pl
from omegaconf import OmegaConf
from pathlib import Path

# Load configuration and initialize
config = OmegaConf.load("./configs/emrqa_9b.yaml")
trainer = pl.Trainer(**config.trainer)
exp_dir = exp_manager(trainer, config.get("exp_manager", None))

# Create QA model
model_qa = nemo_nlp.models.QAModel(cfg=config.model, trainer=trainer)

# Train
trainer.fit(model_qa)

# Test and generate predictions
trainer.test(model_qa)

# Run inference with n-best predictions
output_dir = Path("./results/qa")
output_dir.mkdir(parents=True, exist_ok=True)

model_qa.inference(
    file=config.model.test_ds.file,
    batch_size=16,
    num_samples=-1,
    output_nbest_file=output_dir / "predict_nbest.json",
    output_prediction_file=output_dir / "predict_prediction.json"
)
```

--------------------------------

### Train NLI Model with MedNLI Dataset

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Launches distributed training for the MedNLI task using PyTorch's distributed launch utility. This script fine-tunes a GatorTron model on the MedNLI dataset for Natural Language Inference.

```bash
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-task=8
#SBATCH --cpus-per-task=128
#SBATCH --mem=1000gb

# Paths
DATAROOT=./data/mednli
TRAIN_DATA=${DATAROOT}/mli_train_v1.tsv
VALID_DATA="${DATAROOT}/mli_dev_v1.tsv ${DATAROOT}/mli_test_v1.tsv"
VOCAB_FILE=./vocabs/vocab.txt
PRETRAINED_CHECKPOINT=./checkpoints/gatortron_345m
CHECKPOINT_PATH=./results/mednli_345m

# Distributed launch arguments
DISTRIBUTED_ARGS="--nproc_per_node 8 --nnodes 1 --node_rank 0"

# Launch training (GatorTron-base: L=24, H=1024, A=16)
python -m torch.distributed.launch $DISTRIBUTED_ARGS \
    ./Megatron-LM/tasks/main.py \
    --task MNLI \
    --seed 1234 \
    --keep-last \
    --train-data $TRAIN_DATA \
    --valid-data $VALID_DATA \
    --tokenizer-type BertWordPieceCase \
    --vocab-file $VOCAB_FILE \
    --epochs 10 \
    --pretrained-checkpoint $PRETRAINED_CHECKPOINT \
    --tensor-model-parallel-size 1 \
    --num-layers 24 \
    --hidden-size 1024 \
    --num-attention-heads 16 \
    --micro-batch-size 8 \
    --lr 5.0e-5 \
    --lr-decay-style linear \
    --lr-warmup-fraction 0.065 \
    --seq-length 512 \
    --max-position-embeddings 512 \
    --save-interval 500000 \
    --save $CHECKPOINT_PATH \
    --weight-decay 1.0e-1 \
    --fp16
```

--------------------------------

### Initialize Trainer and Model for STS Task

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Initializes the PyTorch Lightning trainer with NLP DDP plugin and creates a GLUE model for Semantic Textual Similarity (STS) tasks. The trainer is configured for large models and experiment management.

```python
trainer = pl.Trainer(plugins=[NLPDDPPlugin()], **config.trainer)
exp_dir = exp_manager(trainer, config.get("exp_manager", None))

# Create GLUE model for STS task
model = GLUEModel(cfg=config.model, trainer=trainer)

# Train - outputs Pearson/Spearman correlation metrics
trainer.fit(model)
```

--------------------------------

### Build Docker images for Megatron-LM and NeMo

Source: https://github.com/uf-hobi-informatics-lab/gatortron/blob/main/docker/readme.md

Use these commands to build the respective Docker images from local Dockerfiles.

```bash
docker build - < Dockerfile-megatron -t alexgre/megatron
```

```bash
docker build - < Dockerfile-nemo -t alexgre/nemo
```

--------------------------------

### Fine-tune GatorTron for QA

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Command-line interface for fine-tuning GatorTron on extractive question answering tasks.

```python
# Command-line usage
# python qa.py --qa_config ./configs/emrqa_9b.yaml --gpus 8 --pred_output ./results
```

--------------------------------

### Semantic Textual Similarity Fine-tuning

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Configuration and initialization for GLUE-style regression tasks using the STS benchmark.

```yaml
trainer:
  gpus: 8
  max_epochs: 10
  precision: 16
  accelerator: ddp

model:
  task_name: sts-b  # Semantic Textual Similarity benchmark
  dataset:
    data_dir: ./data/sts/clinical_sts
    max_seq_length: 512

  tokenizer:
    tokenizer_name: megatron-bert-cased
    vocab_file: ./vocabs/vocab.txt

  language_model:
    pretrained_model_name: megatron-bert-cased
    lm_checkpoint: ./checkpoints/gatortron_9b/iter_0670000
    config_file: ./checkpoints/gatortron_9b/config.json
```

```python
from nemo.collections.nlp.models import GLUEModel
from nemo.collections.nlp.parts.nlp_overrides import NLPDDPPlugin
from nemo.utils.exp_manager import exp_manager
import pytorch_lightning as pl
from omegaconf import OmegaConf

# Load configuration
config = OmegaConf.load("./configs/sts_9b.yaml")
config.trainer.gpus = 8
```

--------------------------------

### Fine-tune GatorTron for NER

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Configures and trains a TokenClassificationModel for clinical NER tasks using NVIDIA NeMo.

```python
from nemo.collections import nlp as nemo_nlp
from nemo.utils.exp_manager import exp_manager
import pytorch_lightning as pl
from omegaconf import OmegaConf

# Load configuration
config = OmegaConf.load("./configs/i2b2_2010_9b.yaml")
config.trainer.gpus = 8

# Initialize trainer and model
trainer = pl.Trainer(**config.trainer)
exp_dir = exp_manager(trainer, config.get("exp_manager", None))
model_ner = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)

# Train model
trainer.fit(model_ner)
model_ner.save_to("./models/gatortron_ner.nemo")

# Inference on test data
def predict_ner(text_file, model, batch_size=32):
    idx2label = {v: k for k, v in model.cfg.label_ids.items()}
    with open(text_file, "r") as f:
        lines = f.readlines()

    predictions = model._infer(lines, batch_size)
    results = []
    for line, preds in zip(lines, predictions):
        labels = [idx2label[p] for p in preds[:len(line.split())]]
        results.append(" ".join(labels))
    return results

# Evaluate on test set
model_ner.evaluate_from_file(
    output_dir="./results/test",
    text_file="./data/ner/i2b2_2010/text_test.txt",
    labels_file="./data/ner/i2b2_2010/labels_test.txt",
    batch_size=32,
    add_confusion_matrix=True
)
```

--------------------------------

### Convert JSON to Megatron Binary Format

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Preprocesses clinical JSON notes into binary format required for Megatron-LM training.

```bash
python ./Megatron-LM/tools/preprocess_data.py \
    --input ./data/clinical_notes.json \
    --json-keys NOTE_TEXT \
    --split-sentences \
    --tokenizer-type BertWordPieceCase \
    --vocab-file ./vocabs/vocab.txt \
    --output-prefix ./data/bin/clinical_notes \
    --dataset-impl mmap \
    --workers 32 \
    --log-interval 1000
```

--------------------------------

### Load and Use GatorTron with Hugging Face Transformers

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Load GatorTron models and tokenizers from Hugging Face for downstream clinical NLP tasks. The model generates contextual embeddings for text.

```python
from transformers import AutoModel, AutoTokenizer, AutoConfig

# Load GatorTron model and tokenizer (available: gatortron-base, gatortron-medium)
tokenizer = AutoTokenizer.from_pretrained('UFNLP/gatortron-base')
config = AutoConfig.from_pretrained('UFNLP/gatortron-base')
model = AutoModel.from_pretrained('UFNLP/gatortron-base')

# Encode clinical text
clinical_note = "Patient presents with chest pain radiating to left arm. ECG shows ST elevation. Troponin levels elevated."
encoded_input = tokenizer(clinical_note, return_tensors="pt", max_length=512, truncation=True, padding=True)
encoded_output = model(**encoded_input)

# Extract embeddings for downstream tasks
last_hidden_state = encoded_output.last_hidden_state  # Shape: [batch, seq_len, hidden_dim]
pooled_output = encoded_output.pooler_output  # Shape: [batch, hidden_dim] - use for classification

# Example: Feed to NER, RE, or QA task layers
print(f"Sequence embeddings shape: {last_hidden_state.shape}")
print(f"Pooled output shape: {pooled_output.shape}")
```

--------------------------------

### Train Custom Vocabulary with SentencePiece

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Programmatically train a domain-specific BPE vocabulary using SentencePiece on a clinical corpus. The script converts the vocabulary to a BERT-compatible format, including special tokens.

```python
# Example programmatic usage
import sentencepiece as spm
from pathlib import Path

# Train SentencePiece model on clinical corpus
input_corpus = "./clinical_notes.txt"
output_prefix = "./vocabs/gatortron_vocab"
vocab_size = 50000

spm.SentencePieceTrainer.Train(
    f'--input={input_corpus} '
    '--input_format=text '
    f'--model_prefix={output_prefix}/gatortron_vocab '
    f'--vocab_size={vocab_size} '
    '--normalization_rule_name=nmt_nfkc '  # Use nmt_nfkc_cf for lowercase
    '--character_coverage=0.9999 '
    '--model_type=bpe '
    '--train_extremely_large_corpus=true '
    '--max_sentencepiece_length=128 '
    '--max_sentence_length=33536 '
    '--hard_vocab_limit=false '
    '--num_threads=32'
)

# Convert to BERT format: Add special tokens and convert subword markers
bert_special_tokens = ['[PAD]', '[UNK]', '[CLS]', '[SEP]', '[MASK]'] + [f'[unused{i}]' for i in range(1, 100)]
sp_vocab = [line.split('\t')[0] for line in open(f"{output_prefix}/gatortron_vocab.vocab").readlines()]
bert_vocab = bert_special_tokens + [tok.replace('▁', '') if tok.startswith('▁') else '##' + tok for tok in sp_vocab]

with open(f"{output_prefix}/vocab.txt", "w") as f:
    f.write("\n".join(bert_vocab))
```

--------------------------------

### Relation Extraction Fine-tuning

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Configuration and implementation for fine-tuning a text classification model to extract clinical relationships.

```yaml
trainer:
  gpus: 8
  max_epochs: 10
  precision: 16
  accelerator: ddp

model:
  dataset:
    num_classes: 5  # Number of relation types
    max_seq_length: 512

  train_ds:
    file_path: ./data/re/n2c2/train.tsv
    batch_size: 16

  test_ds:
    file_path: ./data/re/n2c2/test.tsv
    batch_size: 32

  tokenizer:
    tokenizer_name: megatron-bert-cased
    vocab_file: ./vocabs/vocab.txt

  language_model:
    pretrained_model_name: megatron-bert-cased
    lm_checkpoint: ./checkpoints/gatortron_9b/iter_0670000
```

```python
from nemo.collections import nlp as nemo_nlp
from nemo.utils.exp_manager import exp_manager
import pytorch_lightning as pl
from omegaconf import OmegaConf
from pathlib import Path

# Load configuration
config = OmegaConf.load("./configs/n2c2_re_9b.yaml")
trainer = pl.Trainer(**config.trainer)
exp_dir = exp_manager(trainer, config.get("exp_manager", None))

# Create text classification model for RE
model = nemo_nlp.models.TextClassificationModel(config.model, trainer=trainer)

# Train
trainer.fit(model)

# Setup test data and evaluate
model.setup_test_data(test_data_config=config.model.test_ds)
trainer.test(model)

# Batch inference on test queries
def load_test_queries(test_file):
    queries = []
    with open(test_file, "r") as f:
        for line in f.readlines():
            queries.append(line.split("\t")[0].strip())
    return queries

queries = load_test_queries(config.model.test_ds.file_path)
predictions = model.classifytext(
    queries,
    batch_size=config.model.test_ds.batch_size,
    max_seq_length=config.model.dataset.max_seq_length
)

# Save predictions
output_path = Path("./results/re")
output_path.mkdir(parents=True, exist_ok=True)
with open(output_path / "predict_labels.txt", "w") as f:
    f.write("\n".join([str(p) for p in predictions]))
```

--------------------------------

### Convert Clinical Notes to Megatron Binary Format

Source: https://context7.com/uf-hobi-informatics-lab/gatortron/llms.txt

Convert clinical notes from CSV/TSV to JSON with sentence tokenization using parallel processing. This is the first step in preparing data for Megatron-LM pretraining.

```bash
# Step 1: Convert CSV/TSV clinical notes to JSON with sentence tokenization
# Uses parallel processing for large datasets

python gatortron_process_data.py \
    --input ./data/clinical_notes.tsv \
    --output ./data/clinical_notes.json \
    --sep $'	' \
    --col_name NOTE_TEXT \
    --cpus 32 \
    --chunk_size 20000
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.