### Initiate Pretraining on Node 1

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Starts the pretraining process on the first node of a two-node setup. This command specifies the node rank, main address, accelerator, devices, and number of nodes, along with the training script and data directories.

```bash
lightning run model \
    --node-rank=0  \
    --main-address=172.16.101.5 \
    --accelerator=cuda \
    --devices=8 \
    --num-nodes=2 \
    pretrain/tinyllama.py --devices 8 --train_data_dir data/slim_star  --val_data_dir data/slim_star
```

--------------------------------

### Initiate Pretraining on Node 2

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Starts the pretraining process on the second node of a two-node setup. Similar to Node 1, it defines the distributed training configuration and points to the pretraining script and data.

```bash
lightning run model \
    --node-rank=1  \
    --main-address=172.16.101.5 \
    --accelerator=cuda \
    --devices=8 \
    --num-nodes=2 \
    pretrain/tinyllama.py --devices 8 --train_data_dir data/slim_star   --val_data_dir data/slim_star
```

--------------------------------

### Initiate Pretraining

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Command to start the pretraining process using Lightning Fabric for multi-GPU/multi-node FSDP training. Configured for `tiny_LLaMA_1b` by default.

```bash
# Single-node 8-GPU pretraining
lightning run model \
    --accelerator=cuda \
    --devices=8 \
    pretrain/tinyllama.py \
    --devices 8 \
    --train_data_dir data/slim_star_combined \
    --val_data_dir data/slim_star_combined
```

--------------------------------

### Install Remaining Dependencies

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Installs the remaining project dependencies using a requirements file and adds tokenizers and sentencepiece.

```bash
pip install -r requirements.txt tokenizers sentencepiece
```

--------------------------------

### Install Flash-Attention 2 and Fused Operators

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Installs Flash-Attention 2 and other fused operators by cloning the repository, building from source, and installing specific components. This process may take several minutes and might display warnings.

```bash
git clone https://github.com/Dao-AILab/flash-attention
cd flash-attention
python setup.py install
cd csrc/rotary && pip install .
cd ../layer_norm && pip install .
cd ../xentropy && pip install .
cd ../..
rm -rf flash-attention
```

--------------------------------

### Two-Node 16-GPU Pretraining Setup

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Configure and run distributed pretraining for TinyLlama across two nodes, each with 8 GPUs. Ensure to run the respective command on each node, specifying the correct node rank.

```bash
lightning run model \
    --node-rank=0 \
    --main-address=172.16.101.5 \
    --accelerator=cuda \
    --devices=8 \
    --num-nodes=2 \
    pretrain/tinyllama.py --devices 8 \
    --train_data_dir data/slim_star \
    --val_data_dir data/slim_star
```

```bash
lightning run model \
    --node-rank=1 \
    --main-address=172.16.101.5 \
    --accelerator=cuda \
    --devices=8 \
    --num-nodes=2 \
    pretrain/tinyllama.py --devices 8 \
    --train_data_dir data/slim_star \
    --val_data_dir data/slim_star
```

--------------------------------

### Install Dependencies

Source: https://github.com/jzhang38/tinyllama/blob/main/chat_gradio/README.md

Install the required Python packages for the Gradio chatbot. Ensure your environment meets the specified version requirements.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install PyTorch Nightly

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Installs the PyTorch nightly build for CUDA 11.8. Ensure CUDA 11.8 is installed prior to running this command.

```bash
pip install --index-url https://download.pytorch.org/whl/nightly/cu118 --pre 'torch>=2.1.0dev'
```

--------------------------------

### Build XFormers from Source

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Builds and installs the XFormers library from its GitHub repository. This is necessary as pre-built binaries for PyTorch 2.1 are not yet available. It first uninstalls and then reinstalls Ninja.

```bash
pip uninstall ninja -y && pip install ninja -U
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
```

--------------------------------

### Evaluating TinyLlama with lm-evaluation-harness

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Run the GPT4All benchmark suite on TinyLlama using the lm-evaluation-harness. Ensure the harness is installed from the master branch and specify the model and tasks.

```bash
# GPT4All benchmark suite (HellaSwag, OpenBookQA, WinoGrande, ARC, BoolQ, PIQA)
pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git@master

python main.py \
    --model hf-causal \
    --model_args pretrained=TinyLlama/TinyLlama-1.1B-Chat-v1.0,dtype="float" \
    --tasks hellaswag,openbookqa,winogrande,arc_easy,arc_challenge,boolq,piqa \
    --device cuda:0 \
    --batch_size 32
# Expected avg acc_norm ~52.30 for Chat-v0.4 (1.5T tokens)
```

--------------------------------

### Load and Combine Packed Datasets

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Demonstrates loading pre-tokenized binary files into `PackedDataset` and combining multiple datasets with specified weights using `CombinedDataset`. Includes DataLoader setup and batch processing.

```python
import glob
from pathlib import Path
from torch.utils.data import DataLoader
from lit_gpt.packed_dataset import PackedDataset, CombinedDataset

data_dir = Path("data/slim_star_combined")
block_size = 2049  # +1 for next-token target

# Build individual datasets for each data source
slim_files  = sorted(glob.glob(str(data_dir / "train_slim*")))
star_files  = sorted(glob.glob(str(data_dir / "train_star*")))

slim_dataset = PackedDataset(
    slim_files,
    n_chunks=8,           # buffer size (controls shuffle quality)
    block_size=block_size,
    shuffle=True,
    seed=42,
    num_processes=1,
    process_rank=0,
)
star_dataset = PackedDataset(
    star_files,
    n_chunks=8,
    block_size=block_size,
    shuffle=True,
    seed=42,
)

# Combine with sampling weights matching original TinyLlama ratio
combined = CombinedDataset(
    datasets=[slim_dataset, star_dataset],
    seed=42,
    weights=[0.693584, 0.306416],
)

loader = DataLoader(combined, batch_size=8, shuffle=False, pin_memory=True)
batch = next(iter(loader))
print(batch.shape)  # torch.Size([8, 2049])

# Use first 2048 tokens as input_ids, last 2048 as targets
input_ids = batch[:, :-1]  # shape: (8, 2048)
targets   = batch[:, 1:]   # shape: (8, 2048)
```

--------------------------------

### Chat Inference with HuggingFace Pipeline

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Perform chat-based text generation using the HuggingFace transformers pipeline. This example demonstrates the v0.x-style prompt format for chat models and configures sampling parameters.

```python
import torch
from transformers import AutoTokenizer, pipeline

model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tokenizer = AutoTokenizer.from_pretrained(model_id)

pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

# v0.x-style prompt format
prompt = "Explain the difference between a list and a tuple in Python."
formatted = f"### Human: {prompt} ### Assistant:"

result = pipe(
    formatted,
    do_sample=True,
    top_k=50,
    top_p=0.9,
    temperature=0.7,
    repetition_penalty=1.1,
    num_return_sequences=1,
    max_new_tokens=512,
)
print(result[0]["generated_text"])
# ### Human: Explain the difference between a list and a tuple in Python.
# ### Assistant: A list is mutable (can be changed after creation) while a tuple
# is immutable. Lists use square brackets [], tuples use parentheses ()...
```

--------------------------------

### Get Default Supported Precision

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Retrieves the default supported precision for training or inference. The output varies based on GPU capabilities.

```python
train_precision = get_default_supported_precision(training=True)
print(train_precision)  # "bf16-mixed" on A100; "16-mixed" on older GPUs

infer_precision = get_default_supported_precision(training=False)
print(infer_precision)  # "bf16-true" on A100
```

--------------------------------

### Load and Create Model Configuration

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Instantiate a configuration for a GPT-style model by name or create a custom one manually. Ensure the model name is registered or all necessary parameters are provided for custom configurations.

```python
from lit_gpt.config import Config

# Load TinyLlama-1.1B config by name
config = Config.from_name("tiny_LLaMA_1b")
print(config.__dict__)
# {
#   'org': 'StatNLP-research', 'name': 'tiny_LLaMA_1b',
#   'block_size': 2048, 'vocab_size': 32000, 'padding_multiple': 64,
#   'padded_vocab_size': 32000, 'n_layer': 22, 'n_head': 32,
#   'n_embd': 2048, 'rotary_percentage': 1.0, 'parallel_residual': False,
#   'bias': False, '_norm_class': 'FusedRMSNorm', 'norm_eps': 1e-05,
#   '_mlp_class': 'LLaMAMLP', 'intermediate_size': 5632, 'n_query_groups': 4
# }

# Build a custom config manually
custom_config = Config(
    block_size=2048,
    vocab_size=32000,
    n_layer=22,
    n_head=32,
    n_embd=2048,
    rotary_percentage=1.0,
    parallel_residual=False,
    bias=False,
    _norm_class="FusedRMSNorm",
    norm_eps=1e-5,
    _mlp_class="LLaMAMLP",
    intermediate_size=5632,
    n_query_groups=4,
)
print(f"Head size: {custom_config.head_size}")  # 64
print(f"Padded vocab size: {custom_config.padded_vocab_size}")  # 32000
```

--------------------------------

### Run Instruct-Eval Benchmarks (MMLU)

Source: https://github.com/jzhang38/tinyllama/blob/main/EVAL.md

Evaluate TinyLlama models on the MMLU benchmark using instruct-eval. Ensure the correct model path is provided.

```bash
CUDA_VISIBLE_DEVICES=0 python main.py mmlu --model_name llama --model_path PY007/TinyLlama-1.1B-intermediate-step-480K-1T
```

--------------------------------

### Run Instruct-Eval Benchmarks (HumanEval)

Source: https://github.com/jzhang38/tinyllama/blob/main/EVAL.md

Evaluate TinyLlama models on the HumanEval benchmark using instruct-eval. Specify the number of samples and the model path.

```bash
CUDA_VISIBLE_DEVICES=3 python main.py humaneval  --model_name llama  --n_sample 1 --model_path PY007/TinyLlama-1.1B-intermediate-step-480K-1T
```

--------------------------------

### Run Instruct-Eval Benchmarks (BBH)

Source: https://github.com/jzhang38/tinyllama/blob/main/EVAL.md

Evaluate TinyLlama models on the BBH benchmark using instruct-eval. Ensure the correct model path is provided.

```bash
CUDA_VISIBLE_DEVICES=1 python main.py bbh --model_name llama --model_path PY007/TinyLlama-1.1B-intermediate-step-480K-1T
```

--------------------------------

### Run Instruct-Eval Benchmarks (DROP)

Source: https://github.com/jzhang38/tinyllama/blob/main/EVAL.md

Evaluate TinyLlama models on the DROP benchmark using instruct-eval. Ensure the correct model path is provided.

```bash
CUDA_VISIBLE_DEVICES=2 python main.py drop --model_name llama --model_path PY007/TinyLlama-1.1B-intermediate-step-480K-1T
```

--------------------------------

### Supervised Fine-Tuning on OpenAssistant Dataset

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Perform supervised fine-tuning on the TinyLlama model using the OpenAssistant (oasst1) dataset. This script utilizes HuggingFace's Seq2SeqTrainer and supports various training configurations.

```bash
python sft/finetune.py \
    --model_name_or_path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --dataset oasst1 \
    --output_dir ./output/tinyllama-oasst \
    --per_device_train_batch_size 16 \
    --gradient_accumulation_steps 1 \
    --learning_rate 2e-4 \
    --max_steps 10000 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type constant \
    --gradient_checkpointing True \
    --do_train \
    --do_eval \
    --save_steps 250
```

--------------------------------

### Download Datasets

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Steps to download SlimPajama and StarCoder datasets using Git LFS and Git clone.

```bash
# Step 1: Download datasets
cd /path/to/dataset
git lfs install
git clone https://huggingface.co/datasets/cerebras/SlimPajama-627B
git clone https://huggingface.co/datasets/bigcode/starcoderdata
```

--------------------------------

### Run GPT4All Benchmarks with lm-eval-harness

Source: https://github.com/jzhang38/tinyllama/blob/main/EVAL.md

Use this command to evaluate TinyLlama models on various GPT4All benchmarks. Specify the model, tasks, and device for evaluation.

```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=PY007/TinyLlama-1.1B-Chat-v0.1,dtype="float" \
    --tasks hellaswag,openbookqa,winogrande,arc_easy,arc_challenge,boolq,piqa\
    --device cuda:0 --batch_size 32
```

--------------------------------

### Download Datasets

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Downloads the SlimPajama and Starcoderdata datasets using Git LFS. Ensure you have sufficient disk space as these datasets are large.

```bash
cd /path/to/dataset
git lfs install
git clone https://huggingface.co/datasets/cerebras/SlimPajama-627B
git clone https://huggingface.co/datasets/bigcode/starcoderdata
```

--------------------------------

### Run Gradio Chat Demo

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Launch a streaming Gradio web interface for the TinyLlama chat model. This demo utilizes TextIteratorStreamer for token-by-token output and formats chat history using a specific template.

```bash
# Install dependencies
pip install gradio transformers torch

# Run the demo (defaults to TinyLlama/TinyLlama-1.1B-Chat-v1.0)
python chat_gradio/app.py
# Launches at http://127.0.0.1:7860
```

--------------------------------

### Model Configuration - Config / Config.from_name

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Demonstrates how to load and use the Config class to define model hyperparameters. You can instantiate a config from a registered model name or build a custom one manually.

```APIDOC
## Config / Config.from_name

### Description
`Config` is a dataclass that defines all architectural hyperparameters for a GPT-style model. `Config.from_name(name)` instantiates a config from a registered model name.

### Usage
```python
from lit_gpt.config import Config

# Load TinyLlama-1.1B config by name
config = Config.from_name("tiny_LLaMA_1b")
print(config.__dict__)

# Build a custom config manually
custom_config = Config(
    block_size=2048,
    vocab_size=32000,
    n_layer=22,
    n_head=32,
    n_embd=2048,
    rotary_percentage=1.0,
    parallel_residual=False,
    bias=False,
    _norm_class="FusedRMSNorm",
    norm_eps=1e-5,
    _mlp_class="LLaMAMLP",
    intermediate_size=5632,
    n_query_groups=4,
)
print(f"Head size: {custom_config.head_size}")
print(f"Padded vocab size: {custom_config.padded_vocab_size}")
```

### Parameters
`Config.from_name(name)`:
- **name** (str): The registered model name to load configuration from.

`Config(...)` (constructor):
- **block_size** (int): The maximum sequence length the model can process.
- **vocab_size** (int): The size of the vocabulary.
- **padding_multiple** (int, optional): Ensures sequence length is a multiple of this value. Defaults to 64.
- **padded_vocab_size** (int, optional): The vocabulary size after padding. Defaults to `vocab_size`.
- **n_layer** (int): The number of transformer layers.
- **n_head** (int): The number of attention heads.
- **n_embd** (int): The dimensionality of the embeddings.
- **rotary_percentage** (float, optional): The percentage of dimensions to apply rotary embeddings to. Defaults to 1.0.
- **parallel_residual** (bool, optional): Whether to use parallel residual connections. Defaults to False.
- **bias** (bool, optional): Whether to use bias terms in linear layers. Defaults to False.
- **_norm_class** (str, optional): The name of the normalization layer class to use. Defaults to "FusedRMSNorm".
- **norm_eps** (float, optional): The epsilon value for the normalization layer. Defaults to 1e-5.
- **_mlp_class** (str, optional): The name of the MLP layer class to use. Defaults to "LLaMAMLP".
- **intermediate_size** (int, optional): The intermediate size of the MLP layer. Defaults to 4 * `n_embd`.
- **n_query_groups** (int, optional): The number of query groups for Grouped Query Attention (GQA). Defaults to `n_head`.

### Returns
- **Config**: An instance of the Config dataclass.
```

--------------------------------

### Resume Pretraining from Checkpoint

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Initiate pretraining for TinyLlama, resuming from a previously saved checkpoint. This command assumes the checkpoint is in the default location or specified elsewhere.

```bash
lightning run model --accelerator=cuda --devices=8 \
    pretrain/tinyllama.py \
    --devices 8 \
    --train_data_dir data/slim_star_combined \
    --resume true
```

--------------------------------

### Instantiate and Use the GPT Model

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Build the full transformer model from a named configuration. The `forward` method supports standard training passes and autoregressive inference with KV caching. Ensure the model is in evaluation mode (`model.eval()`) for inference.

```python
import torch
from lit_gpt.model import GPT
from lit_gpt.config import Config

# Instantiate TinyLlama-1.1B model
model = GPT.from_name("tiny_LLaMA_1b")
model.eval()

# Count parameters
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params:,}")  # ~1,100,000,000

# Standard forward pass (training / prefill)
batch_size, seq_len = 2, 512
input_ids = torch.randint(0, 32000, (batch_size, seq_len))
with torch.no_grad():
    logits = model(input_ids)
print(logits.shape)  # torch.Size([2, 512, 32000])

# Autoregressive forward pass with KV cache (inference)
model.reset_cache()
max_seq_length = 256
prompt_ids = torch.randint(0, 32000, (1, 10))

# Prefill
with torch.no_grad():
    logits = model(prompt_ids, max_seq_length=max_seq_length)

# Single-token decode step using KV cache
input_pos = torch.tensor([10])
next_token = torch.argmax(logits[0, -1]).unsqueeze(0).unsqueeze(0)
with torch.no_grad():
    logits = model(next_token, max_seq_length=max_seq_length, input_pos=input_pos)
print(logits.shape)  # torch.Size([1, 1, 32000])
```

--------------------------------

### Evaluating TinyLlama with Instruct-Eval

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Evaluate TinyLlama on MMLU and HumanEval using Instruct-Eval. Set the CUDA_VISIBLE_DEVICES environment variable and specify the model path.

```bash
# Instruct-Eval (MMLU, BBH, HumanEval, DROP)
CUDA_VISIBLE_DEVICES=0 python main.py mmlu \
    --model_name llama \
    --model_path TinyLlama/TinyLlama-1.1B-Chat-v1.0

CUDA_VISIBLE_DEVICES=0 python main.py humaneval \
    --model_name llama \
    --n_sample 1 \
    --model_path TinyLlama/TinyLlama-1.1B-Chat-v1.0
# Expected HumanEval pass@1 ~9.15 at 2.5T tokens
```

--------------------------------

### Run Gradio Chatbot App

Source: https://github.com/jzhang38/tinyllama/blob/main/chat_gradio/README.md

Execute the Python script to launch the local Gradio interface for the Tinyllama chatbot. Access the provided local URL in your browser to interact.

```bash
python TinyLlama/chat_gradio/app.py
```

--------------------------------

### Core Model - GPT / GPT.from_name

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Shows how to instantiate and use the main transformer model (GPT). It supports features like Grouped Query Attention (GQA), Flash Attention 2, and KV caching for efficient inference.

```APIDOC
## GPT / GPT.from_name

### Description
`GPT` is the main transformer model class. `GPT.from_name(name)` builds the full model from a named config. The `forward` method accepts token indices and optionally `input_pos` for KV-cache-enabled autoregressive inference. The model supports Grouped Query Attention (GQA), Flash Attention 2, fused RoPE embeddings, and optional KV caching.

### Usage
```python
import torch
from lit_gpt.model import GPT
from lit_gpt.config import Config

# Instantiate TinyLlama-1.1B model
model = GPT.from_name("tiny_LLaMA_1b")
model.eval()

# Count parameters
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params:,}")

# Standard forward pass (training / prefill)
batch_size, seq_len = 2, 512
input_ids = torch.randint(0, 32000, (batch_size, seq_len))
with torch.no_grad():
    logits = model(input_ids)
print(logits.shape)

# Autoregressive forward pass with KV cache (inference)
model.reset_cache()
max_seq_length = 256
prompt_ids = torch.randint(0, 32000, (1, 10))

# Prefill
with torch.no_grad():
    logits = model(prompt_ids, max_seq_length=max_seq_length)

# Single-token decode step using KV cache
input_pos = torch.tensor([10])
next_token = torch.argmax(logits[0, -1]).unsqueeze(0).unsqueeze(0)
with torch.no_grad():
    logits = model(next_token, max_seq_length=max_seq_length, input_pos=input_pos)
print(logits.shape)
```

### Parameters
`GPT.from_name(name, **kwargs)`:
- **name** (str): The registered model name to load configuration from.
- **kwargs**: Additional keyword arguments to pass to the `Config` constructor or override config values.

`GPT.forward(self, idx, input_pos=None, *, max_seq_length=None)`:
- **idx** (torch.Tensor): Input token indices of shape `(batch_size, sequence_length)`.
- **input_pos** (torch.Tensor, optional): Current positions in the sequence for KV cache. Defaults to None.
- **max_seq_length** (int, optional): The maximum sequence length to consider for KV cache. Defaults to None.

### Returns
- **GPT**: An instance of the GPT model.
- **torch.Tensor**: Logits of shape `(batch_size, sequence_length, vocab_size)` for standard forward pass, or `(batch_size, 1, vocab_size)` for autoregressive decoding.
```

--------------------------------

### Tokenize Datasets

Source: https://github.com/jzhang38/tinyllama/blob/main/PRETRAIN.md

Tokenizes the downloaded Starcoderdata and SlimPajama datasets using provided scripts. This process divides the data into chunks and requires significant storage space for the processed data.

```bash
python scripts/prepare_starcoder.py --source_path /path/to/starcoderdata/ --tokenizer_path data/llama --destination_path data/slim_star_combined --split train --percentage 1.0
python scripts/prepare_slimpajama.py --source_path /path/to/SlimPajama --tokenizer_path data/llama  --destination_path data/slim_star_combined --split validation --percentage 1.0
python scripts/prepare_slimpajama.py --source_path /path/to/SlimPajama --tokenizer_path data/llama  --destination_path data/slim_star_combined --split train --percentage 1.0
```

--------------------------------

### Auto-precision selection with get_default_supported_precision

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Retrieve the optimal precision string (e.g., 'bf16-mixed', '16-mixed') for the current hardware using `get_default_supported_precision`.

```python
from lit_gpt.utils import get_default_supported_precision


```

--------------------------------

### HuggingFace Assisted Generation with TinyLlama

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Use TinyLlama as a draft model to accelerate inference for larger HuggingFace models. Ensure models are loaded with appropriate data types and device mapping.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

draft_model_id  = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
target_model_id = "TheBloke/guanaco-7B-HF"

tokenizer    = AutoTokenizer.from_pretrained(target_model_id)
draft_model  = AutoModelForCausalLM.from_pretrained(draft_model_id, torch_dtype=torch.float16, device_map="auto")
target_model = AutoModelForCausalLM.from_pretrained(target_model_id, load_in_8bit=True, device_map="auto")

prompt = "### Human: Give me detailed info about Joe Biden. ### Assistant:"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

# Assisted generation uses TinyLlama to propose tokens, target model verifies
outputs = target_model.generate(
    **inputs,
    assistant_model=draft_model,
    max_new_tokens=512,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# Speedup vs native decoding: ~1.8x for guanaco-7b, ~1.8x for guanaco-13b
```

--------------------------------

### Tokenize and Pack SlimPajama Data

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Tokenizes and packs the SlimPajama dataset into the `PackedDataset` format. Supports multi-process parallel tokenization.

```bash
# Step 3: Tokenize and pack SlimPajama (validation split)
python scripts/prepare_slimpajama.py \
    --source_path /path/to/SlimPajama \
    --tokenizer_path data/llama \
    --destination_path data/slim_star_combined \
    --split validation \
    --percentage 1.0
```

```bash
# Step 4: Tokenize and pack SlimPajama (training split, ~1.8T storage)
python scripts/prepare_slimpajama.py \
    --source_path /path/to/SlimPajama \
    --tokenizer_path data/llama \
    --destination_path data/slim_star_combined \
    --split train \
    --percentage 1.0
```

--------------------------------

### Convert HuggingFace Checkpoint to TinyLlama Format

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Converts HuggingFace checkpoints to TinyLlama's `lit_model.pth` format. Automatically detects model families and remaps weights. Saves `lit_config.json`.

```bash
# Convert a downloaded HuggingFace checkpoint
python scripts/convert_hf_checkpoint.py \
    --checkpoint_dir checkpoints/TinyLlama/TinyLlama-1.1B-Chat-v1.0
```

```bash
# Optionally force a specific dtype
python scripts/convert_hf_checkpoint.py \
    --checkpoint_dir checkpoints/meta-llama/Llama-2-7b-hf \
    --model_name Llama-2-7b-hf \
    --dtype bfloat16
```

--------------------------------

### Supervised Fine-Tuning on Custom Local Dataset

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Fine-tune the TinyLlama model on a custom local dataset. The dataset must be in JSON, CSV, or TSV format and contain 'input' and 'output' columns. Specify the dataset path and format accordingly.

```bash
python sft/finetune.py \
    --model_name_or_path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --dataset ./my_data.json \
    --dataset_format input-output \
    --output_dir ./output/tinyllama-custom \
    --max_steps 5000
```

--------------------------------

### Run Llama.cpp Speculative Decoding with C Prompt

Source: https://github.com/jzhang38/tinyllama/blob/main/speculative_decoding/README.md

Execute speculative decoding using Llama.cpp with a large model and a TinyLlama assistant model. This command is suitable for generating C code.

```bash
./speculative \
-m models/CodeLlama-7b-hf/ggml-model-f16.gguf \
-md models/TinyLlama-1.1B-500B-python/ggml-model-q4_0.gguf \
-p "// Quick-sort implementation in C (4 spaces indentation + detailed comments) and sample usage:\n\n#include" \
-e -ngl 1 -t 4 -n 256 -s 20 --temp 0 --draft 8
```

--------------------------------

### llama.cpp Speculative Decoding with TinyLlama

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Utilize TinyLlama as a draft model for llama.cpp's native speculative decoding. Adjust parameters like `-ngl` for GPU layers and `-s` for draft sequence length.

```bash
# llama.cpp speculative decoding (TinyLlama-python as draft for CodeLlama-7b)
./speculative \
    -m  models/CodeLlama-7b-hf/ggml-model-f16.gguf \
    -md models/TinyLlama-1.1B-500B-python/ggml-model-q4_0.gguf \
    -p  "# Quick-sort implementation in Python and sample usage:" \
    -e -ngl 1 -t 4 -n 256 -s 20 --temp 0 --draft 8
# Achieves ~61.5% token acceptance rate; ~33.5 tokens/sec vs ~24 tokens/sec native
```

--------------------------------

### Encode Text to GPU

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Encodes input text into tokens and places them directly onto the specified CUDA device. Shows how to check the device and retrieve tokenizer properties.

```python
import torch
tokens_gpu = tokenizer.encode(text, device=torch.device("cuda"), bos=True)
print(tokens_gpu.device)  # cuda:0

print(f"Vocab size: {tokenizer.vocab_size}")  # 32000
print(f"BOS id: {tokenizer.bos_id}")          # 1
print(f"EOS id: {tokenizer.eos_id}")          # 2
```

--------------------------------

### Run Llama.cpp Speculative Decoding with Python Prompt

Source: https://github.com/jzhang38/tinyllama/blob/main/speculative_decoding/README.md

Execute speculative decoding using Llama.cpp with a large model and a TinyLlama assistant model. This command is suitable for generating Python code.

```bash
./speculative \
-m models/CodeLlama-7b-hf/ggml-model-f16.gguf \
-md models/TinyLlama-1.1B-500B-python/ggml-model-q4_0.gguf \
-p "# Quick-sort implementation in Python and sample usage:" \
-e -ngl 1 -t 4 -n 256 -s 20 --temp 0 --draft 8
```

--------------------------------

### Programmatic Checkpoint Conversion

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Provides programmatic usage for converting HuggingFace checkpoints using the `convert_hf_checkpoint` function. Allows specifying model name and data type.

```python
# Programmatic usage
from pathlib import Path
from scripts.convert_hf_checkpoint import convert_hf_checkpoint
import torch

convert_hf_checkpoint(
    checkpoint_dir=Path("checkpoints/TinyLlama/TinyLlama-1.1B-Chat-v1.0"),
    model_name="tiny_LLaMA_1b",   # must match a name in config.py
    dtype=None,                   # keep original dtype; pass "bfloat16" to convert
)
# Outputs: checkpoints/TinyLlama/TinyLlama-1.1B-Chat-v1.0/lit_model.pth
#          checkpoints/TinyLlama/TinyLlama-1.1B-Chat-v1.0/lit_config.json
```

--------------------------------

### Encode and Decode Text with Tokenizer

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Use the `Tokenizer` class to convert text to token IDs and vice versa. It supports SentencePiece and HuggingFace backends and auto-detects the format. Use `bos=True` and `eos=True` to include special tokens, and `max_length` to cap the output.

```python
from pathlib import Path
from lit_gpt.tokenizer import Tokenizer

# Load from a checkpoint directory containing tokenizer.model (SentencePiece)
tokenizer = Tokenizer(Path("checkpoints/TinyLlama/TinyLlama-1.1B-Chat-v1.0"))

# Encode a string to a token tensor
text = "Hello, TinyLlama!"
tokens = tokenizer.encode(text, bos=True, eos=False)
print(tokens)        # tensor([1, 15043,  29892, 21008, 29931, 12984, 29991])
print(tokens.shape)  # torch.Size([7])

# Encode with a max length cap
tokens_capped = tokenizer.encode(text, bos=True, eos=True, max_length=5)
print(tokens_capped)  # tensor([1, 15043, 29892, 21008, 29931])

# Decode tokens back to string
decoded = tokenizer.decode(tokens)
print(decoded)  # "Hello, TinyLlama!"
```

--------------------------------

### Streaming Chat Inference with Gradio

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Implement streaming chat inference using Gradio, HuggingFace models, and tokenizers. This Python script handles multi-turn conversation formatting and token streaming for a responsive chat interface.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
from threading import Thread

model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model = model.to("cuda" if torch.cuda.is_available() else "cpu")

# Format a multi-turn conversation
history = [["What is Python?", "Python is a high-level programming language."]]
message = "What are its main uses?"

messages = "</s>".join([
    "</s>".join([f"\n<|user|>:{turn[0]}", f"\n<|assistant|>:{turn[1]}"])
    for turn in history + [[message, ""]]
])

inputs = tokenizer([messages], return_tensors="pt").to(model.device)
streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

thread = Thread(target=model.generate, kwargs=dict(
    **inputs, streamer=streamer,
    max_new_tokens=512, do_sample=True,
    top_p=0.95, top_k=50, temperature=0.7,
))
thread.start()

output = ""
for token in streamer:
    output += token
    print(token, end="", flush=True)

```

--------------------------------

### Memory-efficient loss computation with chunked_cross_entropy

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Employ `chunked_cross_entropy` for memory-efficient loss calculation during fine-tuning with long sequences. Set `chunk_size=0` for standard cross-entropy behavior.

```python
import torch
from lit_gpt.utils import chunked_cross_entropy

logits  = torch.randn(2, 2048, 32000)   # (batch, seq_len, vocab)
targets = torch.randint(0, 32000, (2, 2048))

# Chunked computation (chunk_size=128 by default)
loss = chunked_cross_entropy(logits, targets, chunk_size=128)
print(loss)  # tensor(10.37...)   (same result as F.cross_entropy)

# No chunking (equivalent to standard cross entropy)
loss_full = chunked_cross_entropy(logits, targets, chunk_size=0)
print(torch.allclose(loss, loss_full, atol=1e-4))  # True
```

--------------------------------

### Tokenize and Pack StarCoder Data

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Tokenizes and packs the StarCoder dataset into the `PackedDataset` format. Uses multi-process parallel tokenization.

```bash
# Step 2: Tokenize and pack StarCoder (training split)
python scripts/prepare_starcoder.py \
    --source_path /path/to/starcoderdata/ \
    --tokenizer_path data/llama \
    --destination_path data/slim_star_combined \
    --split train \
    --percentage 1.0
```

--------------------------------

### Memory-efficient checkpoint loading with lazy_load

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Use the `lazy_load` context manager to inspect PyTorch checkpoints without loading all weights into RAM. Tensors are materialized only when accessed.

```python
from pathlib import Path
from lit_gpt.utils import lazy_load

# Inspect a checkpoint without loading all weights into RAM
with lazy_load(Path("checkpoints/TinyLlama/TinyLlama-1.1B-Chat-v1.0/lit_model.pth")) as sd:
    print(list(sd.keys())[:5])
    # ['transformer.wte.weight', 'transformer.h.0.norm_1.weight', ...]

    # Tensor metadata is available without loading
    print(sd["lm_head.weight"].shape)  # torch.Size([32000, 2048])

    # Tensor is loaded only when data is accessed
    weight = sd["transformer.wte.weight"]._load_tensor()
    print(weight.dtype)   # torch.float16
    print(weight.shape)   # torch.Size([32000, 2048])
```

--------------------------------

### Count Model Parameters

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Counts the total, trainable, and frozen parameters of a given PyTorch model. Requires importing `num_parameters` from `lit_gpt.utils`.

```python
import torch.nn as nn
from lit_gpt.model import GPT
from lit_gpt.utils import num_parameters

model = GPT.from_name("tiny_LLaMA_1b")

total     = num_parameters(model)
trainable = num_parameters(model, requires_grad=True)
frozen    = num_parameters(model, requires_grad=False)

print(f"Total:     {total:,}")      # 1,100,048,384
print(f"Trainable: {trainable:,}")  # 1,100,048,384
print(f"Frozen:    {frozen:,}")     # 0
```

--------------------------------

### Tokenizer - Tokenizer

Source: https://context7.com/jzhang38/tinyllama/llms.txt

Provides a unified interface for tokenizing and decoding text using either SentencePiece or HuggingFace backends. It automatically detects the tokenizer type from the checkpoint.

```APIDOC
## Tokenizer

### Description
`Tokenizer` is a unified wrapper that supports both SentencePiece (`.model`) and HuggingFace (`tokenizer.json`) backends. It auto-detects the backend from the checkpoint directory. Provides `encode()` (string → token tensor) and `decode()` (token tensor → string).

### Usage
```python
from pathlib import Path
from lit_gpt.tokenizer import Tokenizer

# Load from a checkpoint directory containing tokenizer.model (SentencePiece)
tokenizer = Tokenizer(Path("checkpoints/TinyLlama/TinyLlama-1.1B-Chat-v1.0"))

# Encode a string to a token tensor
text = "Hello, TinyLlama!"
tokens = tokenizer.encode(text, bos=True, eos=False)
print(tokens)
print(tokens.shape)

# Encode with a max length cap
tokens_capped = tokenizer.encode(text, bos=True, eos=True, max_length=5)
print(tokens_capped)

# Decode tokens back to string
decoded = tokenizer.decode(tokens)
print(decoded)
```

### Parameters
`Tokenizer(checkpoint_dir, **kwargs)`:
- **checkpoint_dir** (Path): Path to the checkpoint directory containing the tokenizer file (`tokenizer.model` or `tokenizer.json`).
- **kwargs**: Additional keyword arguments to pass to the underlying tokenizer backend.

`encode(self, text, bos=False, eos=False, max_length=None)`:
- **text** (str): The input string to encode.
- **bos** (bool, optional): Whether to prepend the beginning-of-sequence token. Defaults to False.
- **eos** (bool, optional): Whether to append the end-of-sequence token. Defaults to False.
- **max_length** (int, optional): The maximum length of the output token tensor. If exceeded, the sequence is truncated. Defaults to None.

`decode(self, tokens)`:
- **tokens** (torch.Tensor): A tensor of token IDs to decode.

### Returns
- **Tokenizer**: An instance of the Tokenizer class.
- **torch.Tensor**: Encoded token IDs.
- **str**: Decoded string.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.