### Basic Installation and Setup

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/cheat_sheet.md.txt

Commands for installing the extension and importing it into a Python environment.

```bash
python -m pip install intel_extension_for_pytorch
```

```python
import intel_extension_for_pytorch as ipex
```

--------------------------------

### Install Intel® Extension for PyTorch* in Develop Mode

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Comparison of standard installation versus develop mode installation for active development.

```bash
python setup.py install
```

```bash
python setup.py develop
```

--------------------------------

### Install Documentation Prerequisites

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Installs the necessary Python packages for building documentation. Ensure you are in the 'docs' directory.

```bash
cd docs
pip install -r requirements.txt
```

--------------------------------

### Kernel Implementation File Example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/isa_dynamic_dispatch.html

This is an example of a kernel implementation file that will be copied and modified for different ISA levels.

```cpp
csrc/cpu/aten/kernels/AdaptiveAveragePoolingKrnl.cpp
```

--------------------------------

### Install Intel® Extension for PyTorch*

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/cheat_sheet.html

Use this command to install the extension via pip.

```bash
python -m pip install intel_extension_for_pytorch
```

--------------------------------

### Import FSDP-related packages for example

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/features/FSDP.html

Import all necessary libraries for the FSDP example, including PyTorch, extensions, optimizers, datasets, and distributed components.

```python
"""
Import Intel® extension for Pytorch* and Intel® oneCCL Bindings for Pytorch*
"""
import os
import argparse
import functools
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

# Import Intel® extension for Pytorch* and Intel® oneCCL Bindings for Pytorch*
import intel_extension_for_pytorch
import oneccl_bindings_for_pytorch

from torchvision import datasets, transforms
from torch.optim.lr_scheduler import StepLR

import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.fully_sharded_data_parallel import (
    CPUOffload,
    BackwardPrefetch,
)
from torch.distributed.fsdp.wrap import (
    size_based_auto_wrap_policy,
    enable_wrap,
    wrap,
)
```

--------------------------------

### DDP with MPI Launcher on Single Node

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DDP.md.txt

This example demonstrates how to use MPI as the launcher to start Distributed Data Parallel (DDP) training on a single node with multiple devices. It requires setting environment variables for rank and world size, initializing the process group with the 'ccl' backend, and then proceeding with model definition and training loop.

```python
import os
import torch
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel as DDP
import torch.distributed as dist
import intel_extension_for_pytorch
import oneccl_bindings_for_pytorch


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = nn.Linear(4, 5)

    def forward(self, input):
        return self.linear(input)


if __name__ == "__main__":

    torch.xpu.manual_seed(123)  # set a seed number
    mpi_world_size = int(os.environ.get('PMI_SIZE', -1))
    mpi_rank = int(os.environ.get('PMI_RANK', -1))
    if mpi_world_size > 0:
        os.environ['RANK'] = str(mpi_rank)
        os.environ['WORLD_SIZE'] = str(mpi_world_size)
    else:
        # set the default rank and world size to 0 and 1
        os.environ['RANK'] = str(os.environ.get('RANK', 0))
        os.environ['WORLD_SIZE'] = str(os.environ.get('WORLD_SIZE', 1))
    os.environ['MASTER_ADDR'] = '127.0.0.1'  # your master address
    os.environ['MASTER_PORT'] = '29500'  # your master port

    # Initialize the process group with ccl backend
    dist.init_process_group(backend='ccl')

    # For single-node distributed training, local_rank is the same as global rank
    local_rank = dist.get_rank()
    # Only set device for distributed training on GPU
    device = "xpu:{}".format(local_rank)
    model = Model().to(device)
    if dist.get_world_size() > 1:
        model = DDP(model, device_ids=[device])

    optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
    loss_fn = nn.MSELoss().to(device)
    for i in range(3):
        print("Runing Iteration: {} on device {}".format(i, device))
        input = torch.randn(2, 4).to(device)
        labels = torch.randn(2, 5).to(device)
        # forward
        print("Runing forward: {} on device {}".format(i, device))
        res = model(input)
        # loss
        print("Runing loss: {} on device {}".format(i, device))
        L = loss_fn(res, labels)
        # backward
        print("Runing backward: {} on device {}".format(i, device))
        L.backward()
        # update
        print("Runing optim: {} on device {}".format(i, device))
        optimizer.step()
```

```bash
mpirun -n 2 -l python Example_DDP.py
```

--------------------------------

### Verify CMake Installation

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Ensure your CMake installation is working correctly by compiling and running a simple 'Hello World' C program. This is a prerequisite for installing Intel® Extension for PyTorch*.

```bash
printf '#include <stdio.h>\nint main() { printf("Hello World");}'|clang -x c -; ./a.out
```

--------------------------------

### HyperTune Configuration YAML Example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/hypertune.html

Define hyperparameters, their search spaces, and tuning strategy in a YAML file for HyperTune. This example shows settings for tuning launcher hyperparameters.

```yaml
tuning:
  strategy: grid
  max_trials: 100

output_dir: /path/to/saving/directory

hyperparams:
  launcher:
    hp: ['ncores_per_instance', 'ninstances']
    ncores_per_instance: all_physical_cores
    ninstances:  [1]

```

--------------------------------

### Install oneccl_bindings_for_pytorch from Prebuilt Wheels

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/features/DDP.html

Install the oneccl_bindings_for_pytorch package using pip with a specified repository URL. Replace <REPO_URL> with the appropriate URL for CPU or GPU.

```bash
python -m pip install oneccl_bind_pt --extra-index-url <REPO_URL>
```

--------------------------------

### Full Horovod Training Example

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/horovod.md.txt

A complete example demonstrating dataset partitioning, distributed optimization, and parameter broadcasting.

```python
import torch
import intel_extension_for_pytorch
import horovod.torch as hvd

# Initialize Horovod
hvd.init()

# Pin GPU to be used to process local rank (one GPU per process)
devid = hvd.local_rank()
torch.xpu.set_device(devid)
device = "xpu:{}".format(devid)

# Define dataset...
train_dataset = ...

# Partition dataset among workers using DistributedSampler
train_sampler = torch.utils.data.distributed.DistributedSampler(
    train_dataset, num_replicas=hvd.size(), rank=hvd.rank())

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=..., sampler=train_sampler)

# Build model...
model = ...
model.to(device)

optimizer = optim.SGD(model.parameters())

# Add Horovod Distributed Optimizer
optimizer = hvd.DistributedOptimizer(optimizer, named_parameters=model.named_parameters())

# Broadcast parameters from rank 0 to all other processes.
hvd.broadcast_parameters(model.state_dict(), root_rank=0)

for epoch in range(100):
   for batch_idx, (data, target) in enumerate(train_loader):
       optimizer.zero_grad()
       output = model(data)
       loss = F.nll_loss(output, target)
       loss.backward()
       optimizer.step()
       if batch_idx % args.log_interval == 0:
           print('Train Epoch: {} [{}/{}]	Loss: {}'.format(
               epoch, batch_idx * len(data), len(train_sampler), loss.item()))
```

--------------------------------

### Import Example Dependencies

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/FSDP.md.txt

Standard imports for a PyTorch FSDP training script.

```python
"""
Import Intel® extension for Pytorch\* and Intel® oneCCL Bindings for Pytorch\*
"""
import os
import argparse
import functools
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
```

--------------------------------

### Example log file names

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/launch_script.md.txt

Example output filenames generated when logging is enabled via the --log-dir knob.

```text
run_20210712212258_instances.log
run_20210712212258_instance_0_cores_0-43.log
```

--------------------------------

### Start TorchServe with Intel Extension for PyTorch

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/torchserve.md.txt

Command to initiate the TorchServe process with the required configuration file.

```bash
torchserve --start --ncs --model-store model_store --ts-config config.properties
```

--------------------------------

### Start TorchServe with IPEX Enabled

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/performance_tuning/torchserve.html

Starts TorchServe with Intel® Extension for PyTorch* enabled. Ensure that `ipex_enable=true` is set in the `config.properties` file.

```bash
torchserve --start --ncs --model-store model_store --ts-config config.properties

```

--------------------------------

### Hypertune Output Example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/hypertune.html

Example output from the Hypertune process, showing the best configuration found and the corresponding minimum latency achieved.

```text
Best configuration found is: {'ncores_per_instance': 15, 'ninstances': 1, 'use_all_nodes': True, 'use_logical_cores': False, 'disable_numactl': False, 'disable_iomp': False, 'malloc': 'tc'}
latency: 12.339081764221191
```

--------------------------------

### Binary Installation of Intel Extension for PyTorch

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/releases.md.txt

Install Intel Extension for PyTorch version 1.9.0 using pip with a specific find-links URL.

```bash
python -m pip install torch_ipex==1.9.0 -f https://software.intel.com/ipex-whl-stable
```

--------------------------------

### Compilation Command for C++ IPEX Example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/examples.html

Builds the C++ inference application using CMake. Ensure the CMAKE_PREFIX_PATH is set to your PyTorch installation directory.

```bash
$ cd examples/cpu/inference/cpp
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=<LIBPYTORCH_PATH> ..
$ make


```

--------------------------------

### Vanilla setuptools Extension Equivalent

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/features/DPC%2B%2B_Extension.html

This shows the equivalent configuration using vanilla setuptools.Extension, highlighting the necessary include paths and language setting for DPC++.

```python
Extension(
   name='lltm_xpu',
   sources=['lltm_xpu.cpp', 'lltm_xpu_kernel.cpp',],
   include_dirs=cpp_extension.include_paths(),
   language='c++')

```

--------------------------------

### Fast BERT Usage Example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/fast_bert.html

Demonstrates how to use the `ipex.fast_bert` API to optimize a BERT model for training. Ensure you have the correct version of the Transformers library installed.

```python
import torch
from transformers import BertModel

model = BertModel.from_pretrained("bert-base-uncased", attn_implementation="eager")
model.eval()

vocab_size = model.config.vocab_size
batch_size = 1
seq_length = 512
data = torch.randint(vocab_size, size=[batch_size, seq_length])
torch.manual_seed(43)

#################### code changes ####################  # noqa F401
import intel_extension_for_pytorch as ipex

model = ipex.fast_bert(model, dtype=torch.bfloat16)
######################################################  # noqa F401

with torch.no_grad():
    model(data)

print("Execution finished")

```

--------------------------------

### Get XPU Device Information

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DDP.md.txt

Retrieve the XPU device string using the `local_rank` obtained from the distributed setup. This string is used to specify the device for model training.

```python
xpu = "xpu:{}".format(local_rank)
print("DDP Use XPU: {} for training".format(xpu))
```

--------------------------------

### Single-Instance Training Setup

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/examples.html

Illustrates the essential code changes for setting up single-instance training with Intel® Extension for PyTorch*, including importing the library, optimizing the model and optimizer, and moving data to the XPU.

```python
import torch
import intel_extension_for_pytorch as ipex

model = Model()
criterion = ...
optimizer = ...
model.train()
# Move model and loss criterion to xpu before calling ipex.optimize()
model = model.to("xpu")
criterion = criterion.to("xpu")

# For Float32
model, optimizer = ipex.optimize(model, optimizer=optimizer)
# For BFloat16
model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.bfloat16)

dataloader = ...
for (input, target) in dataloader:
    input = input.to("xpu")
    target = target.to("xpu")
    optimizer.zero_grad()
    # For Float32
    output = model(input)

    # For BFloat16
    with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
        output = model(input)

    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
```

--------------------------------

### Example of Configuring Core Binding

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/runtime_extension.html

Shows how to bind a CPU Pool to physical cores using the `ipex.cpu.runtime.pin` context manager for performance optimization without using asynchronous tasks.

```python
cpu_pool = ipex.cpu.runtime.CPUPool(node_id=0)
with ipex.cpu.runtime.pin(cpu_pool):
    y_runtime = traced_model1(x)
```

--------------------------------

### LLM Optimization with ipex.llm.optimize

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/getting_started.md.txt

Optimize Large Language Models (LLMs) using `ipex.llm.optimize`. This example shows setup for model loading, tokenization, and applying optimizations for specified data types.

```python
import torch
import intel_extension_for_pytorch as ipex
import argparse
from transformers import (
    AutoConfig,
    AutoModelForCausalLM,
    AutoTokenizer,
)

# args
parser = argparse.ArgumentParser("Generation script (fp32/bf16 path)", add_help=False)
parser.add_argument(
    "--dtype",
    type=str,
    choices=["float32", "bfloat16"],
    default="float32",
    help="choose the weight dtype and whether to enable auto mixed precision or not",
)
parser.add_argument(
    "--max-new-tokens", default=32, type=int, help="output max new tokens"
)
parser.add_argument(
    "--prompt", default="What are we having for dinner?", type=str, help="input prompt"
)
parser.add_argument("--greedy", action="store_true")
parser.add_argument("--batch-size", default=1, type=int, help="batch size")
args = parser.parse_args()
print(args)

# dtype
amp_enabled = True if args.dtype != "float32" else False
amp_dtype = getattr(torch, args.dtype)

# load model
model_id = MODEL_ID
config = AutoConfig.from_pretrained(
    model_id, torchscript=True, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=amp_dtype,
    config=config,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True
)
model = model.eval()
model = model.to(memory_format=torch.channels_last)

# Intel(R) Extension for PyTorch*
model = ipex.llm.optimize(
    model,
    dtype=amp_dtype,
    inplace=True,
    deployment_mode=True,
)

# generate args
num_beams = 1 if args.greedy else 4
generate_kwargs = dict(do_sample=False, temperature=0.9, num_beams=num_beams)

# input prompt
prompt = args.prompt
input_size = tokenizer(prompt, return_tensors="pt").input_ids.size(dim=1)
print("---- Prompt size:", input_size)
prompt = [prompt] * args.batch_size
```

--------------------------------

### Dynamic Quantization Example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/examples.html

This snippet demonstrates how to perform dynamic quantization on a Hugging Face Transformers model using Intel Extension for PyTorch. It includes importing necessary libraries, preparing the model, converting it to INT8, tracing, freezing, and saving the quantized model. Ensure the 'transformers' package is installed.

```python
import torch
import intel_extension_for_pytorch as ipex
from intel_extension_for_pytorch.quantization import prepare, convert
from transformers import BertModel

model = BertModel.from_pretrained("bert-base-uncased")
model.eval()

vocab_size = model.config.vocab_size
batch_size = 128
seq_length = 512
data = torch.randint(vocab_size, size=[batch_size, seq_length])

qconfig_mapping = ipex.quantization.default_dynamic_qconfig_mapping
prepared_model = prepare(model, qconfig_mapping, example_inputs=data)

converted_model = convert(prepared_model)
with torch.no_grad():
    traced_model = torch.jit.trace(
        converted_model, (data,), check_trace=False, strict=False
    )
    traced_model = torch.jit.freeze(traced_model)

traced_model.save("dynamic_quantized_model.pt")

print("Saved model to: dynamic_quantized_model.pt")
```

--------------------------------

### Example Log File Names

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/performance_tuning/launch_script.html

Illustrates the naming convention for log files generated by the launch script, showing instance logs and core-specific logs.

```text
run_20210712212258_instances.log
run_20210712212258_instance_0_cores_0-43.log
```

--------------------------------

### Vanilla setuptools Extension

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DPC%2B%2B_Extension.md.txt

This is the equivalent vanilla setuptools code for defining a C++ extension, specifying source files, include directories, and language.

```python
Extension(
   name='lltm_xpu',
   sources=['lltm_xpu.cpp', 'lltm_xpu_kernel.cpp',],
   include_dirs=cpp_extension.include_paths(),
   language='c++')
```

--------------------------------

### Convert to Static Quantized Model and Deploy

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/int8_overview.html

Convert the prepared model to a static quantized model and then trace and freeze it for inference or save it for deployment. Ensure example_inputs match the real input size.

```python
# make sure the example_inputs's size is same as the real input's size 
convert_model = convert(prepared_model)
with torch.no_grad():
    traced_model = torch.jit.trace(convert_model, example_input)
    traced_model = torch.jit.freeze(traced_model)
# for inference 
y = traced_model(x)

# or save the model to deploy

# traced_model.save("quantized_model.pt")
# quantized_model = torch.jit.load("quantized_model.pt")
# quantized_model = torch.jit.freeze(quantized_model.eval())
# ...

```

--------------------------------

### Install Intel Extension for PyTorch 1.9.0

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/releases.html

Use this command to install version 1.9.0 of the Intel Extension for PyTorch, which supports binary installation.

```bash
python -m pip install torch_ipex==1.9.0 -f https://software.intel.com/ipex-whl-stable
```

--------------------------------

### Install Triton for XPU

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/torch_compile_gpu.md.txt

Install the specific version of PyTorch-Triton for XPU using the provided command. Ensure to remove any existing '~/.triton' cache files before installation to avoid conflicts.

```Bash
pip install --pre pytorch-triton-xpu==3.1.0+91b14bf559 --index-url https://download.pytorch.org/whl/nightly/xpu
```

--------------------------------

### Install Intel Optimization for Horovod

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/horovod.md.txt

Use pip to install the optimized Horovod package.

```bash
python -m pip install intel-optimization-for-horovod
```

--------------------------------

### Instance Launch Log Details

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/performance_tuning/launch_script.html

This log file contains detailed information about the execution launch, including environment variables, thread settings, and the specific `numactl` commands used for each instance.

```bash
$ cat logs/run_20210712221415_instances.log
2021-07-12 22:14:15,140 - __main__ - WARNING - Both TCMalloc and JeMalloc are not found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /home/<user>/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance
2021-07-12 22:14:15,140 - __main__ - INFO - OMP_NUM_THREADS=4
2021-07-12 22:14:15,140 - __main__ - INFO - Using Intel OpenMP
2021-07-12 22:14:15,140 - __main__ - INFO - KMP_AFFINITY=granularity=fine,compact,1,0
2021-07-12 22:14:15,140 - __main__ - INFO - KMP_BLOCKTIME=1
2021-07-12 22:14:15,140 - __main__ - INFO - LD_PRELOAD=<VIRTUAL_ENV>/lib/libiomp5.so
2021-07-12 22:14:15,140 - __main__ - INFO - numactl -C 0-3 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_0_cores_0-3.log
2021-07-12 22:14:15,143 - __main__ - INFO - numactl -C 4-7 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_1_cores_4-7.log
2021-07-12 22:14:15,146 - __main__ - INFO - numactl -C 8-11 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_2_cores_8-11.log
2021-07-12 22:14:15,149 - __main__ - INFO - numactl -C 12-15 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_3_cores_12-15.log
2021-07-12 22:14:15,151 - __main__ - INFO - numactl -C 16-19 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_4_cores_16-19.log
2021-07-12 22:14:15,154 - __main__ - WARNING - Numa Aware: cores:['20', '21', '22', '23'] on different NUMA nodes
2021-07-12 22:14:15,154 - __main__ - INFO - numactl -C 20-23 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_5_cores_20-23.log
2021-07-12 22:14:15,157 - __main__ - INFO - numactl -C 24-27 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_6_cores_24-27.log
2021-07-12 22:14:15,159 - __main__ - INFO - numactl -C 28-31 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_7_cores_28-31.log
2021-07-12 22:14:15,162 - __main__ - INFO - numactl -C 32-35 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_8_cores_32-35.log
2021-07-12 22:14:15,164 - __main__ - INFO - numactl -C 36-39 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_9_cores_36-39.log
2021-07-12 22:14:15,167 - __main__ - INFO - numactl -C 40-43 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_10_cores_40-43.log

```

--------------------------------

### Install Linting Dependencies

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Install the necessary dependencies for local linting checks. This command should typically be run only once.

```bash
make setup_lint
```

--------------------------------

### Build DPC++ Extension with setuptools

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DPC%2B%2B_Extension.md.txt

Use DPCPPExtension and DpcppBuildExtension for building DPC++ extensions via setuptools. Ensure correct source files and extension name are provided.

```python
from setuptools import setup
import torch
import intel_extension_for_pytorch
from torch.xpu.cpp_extension import DPCPPExtension, DpcppBuildExtension

setup(
    name='lltm',
    ext_modules=[
        DPCPPExtension('lltm_xpu', [
            'lltm_xpu.cpp',
            'lltm_xpu_kernel.cpp',
        ])
    ],
    cmdclass={
        'build_ext': DpcppBuildExtension
    })
```

--------------------------------

### Install oneMKL via Conda

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/known_issues.md.txt

Install MKL libraries from Conda to resolve undefined symbol errors during build.

```bash
conda install mkl
conda install mkl-include
```

--------------------------------

### Install oneCCL Bindings via pip

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DDP.md.txt

Install the oneCCL bindings package using the specified repository URL.

```bash
python -m pip install oneccl_bind_pt --extra-index-url <REPO_URL>
```

--------------------------------

### Example Log Output for GNU OpenMP

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/launch_script.md.txt

Sample log output confirming the configuration of OMP_SCHEDULE, OMP_PROC_BIND, and the execution command.

```text
2021-07-13 15:25:00,760 - __main__ - WARNING - Both TCMalloc and JeMalloc are not found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /home/<user>/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance
2021-07-13 15:25:00,761 - __main__ - INFO - OMP_SCHEDULE=STATIC
2021-07-13 15:25:00,761 - __main__ - INFO - OMP_PROC_BIND=CLOSE
2021-07-13 15:25:00,761 - __main__ - INFO - OMP_NUM_THREADS=44
2021-07-13 15:25:00,761 - __main__ - WARNING - Numa Aware: cores:['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43'] on different NUMA nodes
2021-07-13 15:25:00,761 - __main__ - INFO - numactl -C 0-43 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210713152500_instance_0_cores_0-43.log
```

--------------------------------

### Install Triton for XPU

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/features/torch_compile_gpu.html

Install the specified version of PyTorch-Triton for XPU. Ensure you are using the nightly build index URL.

```bash
pip install --pre pytorch-triton-xpu==3.1.0+91b14bf559 --index-url https://download.pytorch.org/whl/nightly/xpu
```

--------------------------------

### Initialize FSDP Distributed Training

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/FSDP.md.txt

Main entry point for FSDP training, setting up datasets and distributed samplers for XPU.

```python
"""
Change the device related logic from 'rank' to '"xpu:{}".format(rank)'.
Specify the argument `device_ids` as XPU device ("xpu:{}".format(rank)) in FSDP API.
"""
def fsdp_main(rank, world_size, args):
    setup(rank, world_size)

    transform=transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))
    ])

    dataset1 = datasets.MNIST('../data', train=True, download=True,
                        transform=transform)
    dataset2 = datasets.MNIST('../data', train=False,
                        transform=transform)

    sampler1 = DistributedSampler(dataset1, rank=rank, num_replicas=world_size, shuffle=True)
    sampler2 = DistributedSampler(dataset2, rank=rank, num_replicas=world_size)

    train_kwargs = {'batch_size': args.batch_size, 'sampler': sampler1}
```

--------------------------------

### Performance Configuration

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/cheat_sheet.md.txt

Commands to run launch scripts and hyperparameter tuning.

```bash
ipexrun [knobs] <your_pytorch_script> [args]
```

```bash
python -m intel_extension_for_pytorch.cpu.hypertune --conf-file <your_conf_file> <your_python_script> [args]
```

--------------------------------

### Build DPC++ Extension with setuptools

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/features/DPC%2B%2B_Extension.html

Use this setup.py script with setuptools to compile DPC++ extensions ahead of time (AOT). It defines the extension name and source files.

```python
from setuptools import setup
import torch
import intel_extension_for_pytorch
from torch.xpu.cpp_extension import DPCPPExtension, DpcppBuildExtension

setup(
    name='lltm',
    ext_modules=[
        DPCPPExtension('lltm_xpu', [
            'lltm_xpu.cpp',
            'lltm_xpu_kernel.cpp',
        ])
    ],
    cmdclass={
        'build_ext': DpcppBuildExtension
    })

```

--------------------------------

### Analyze Instance Launch Log

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/launch_script.md.txt

The main instances log file contains crucial information about the execution environment, including warnings about missing memory allocators and details on OpenMP settings and NUMA affinity.

```bash
$ cat logs/run_20210712221415_instances.log
2021-07-12 22:14:15,140 - __main__ - WARNING - Both TCMalloc and JeMalloc are not found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /home/<user>/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance
2021-07-12 22:14:15,140 - __main__ - INFO - OMP_NUM_THREADS=4
2021-07-12 22:14:15,140 - __main__ - INFO - Using Intel OpenMP
2021-07-12 22:14:15,140 - __main__ - INFO - KMP_AFFINITY=granularity=fine,compact,1,0
2021-07-12 22:14:15,140 - __main__ - INFO - KMP_BLOCKTIME=1
2021-07-12 22:14:15,140 - __main__ - INFO - LD_PRELOAD=<VIRTUAL_ENV>/lib/libiomp5.so
2021-07-12 22:14:15,140 - __main__ - INFO - numactl -C 0-3 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_0_cores_0-3.log
2021-07-12 22:14:15,143 - __main__ - INFO - numactl -C 4-7 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_1_cores_4-7.log
2021-07-12 22:14:15,146 - __main__ - INFO - numactl -C 8-11 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_2_cores_8-11.log
2021-07-12 22:14:15,149 - __main__ - INFO - numactl -C 12-15 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_3_cores_12-15.log
2021-07-12 22:14:15,151 - __main__ - INFO - numactl -C 16-19 -m 0 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_4_cores_16-19.log
2021-07-12 22:14:15,154 - __main__ - WARNING - Numa Aware: cores:['20', '21', '22', '23'] on different NUMA nodes
2021-07-12 22:14:15,154 - __main__ - INFO - numactl -C 20-23 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_5_cores_20-23.log
2021-07-12 22:14:15,157 - __main__ - INFO - numactl -C 24-27 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_6_cores_24-27.log
2021-07-12 22:14:15,159 - __main__ - INFO - numactl -C 28-31 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_7_cores_28-31.log
2021-07-12 22:14:15,162 - __main__ - INFO - numactl -C 32-35 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_8_cores_32-35.log
2021-07-12 22:14:15,164 - __main__ - INFO - numactl -C 36-39 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_9_cores_36-39.log
2021-07-12 22:14:15,167 - __main__ - INFO - numactl -C 40-43 -m 1 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712221415_instance_10_cores_40-43.log
```

--------------------------------

### Install Latest Intel Extension for PyTorch

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/releases.html

Install the latest stable version of the Intel Extension for PyTorch directly from PyPI using pip.

```python
pip install intel_extension_for_pytorch
```

--------------------------------

### Enable TorchServe with Launcher (Default Configuration)

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/performance_tuning/torchserve.html

Add these lines to `config.properties` to enable the launcher with its default settings for TorchServe.

```properties
ipex_enable=true
cpu_launcher_enable=true

```

--------------------------------

### Clean and Reinstall Intel® Extension for PyTorch*

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

If you encounter errors during `python setup.py develop`, clean your build directory and repository. This involves removing the build folder, cleaning untracked files, and re-initializing submodules before reinstalling.

```bash
git submodule deinit -f .
git clean -xdf
python setup.py clean
git submodule update --init --recursive --jobs 0 # very important to sync the submodules
python setup.py develop                          # then try running the command again
```

--------------------------------

### Enable Launcher with Logical Cores

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/torchserve.md.txt

Configure `config.properties` to use the CPU launcher with all cores, including logical ones.

```properties
ipex_enable=true
cpu_launcher_enable=true
cpu_launcher_args=--use_logical_core
```

--------------------------------

### Install Jemalloc using Conda

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/tuning_guide.md.txt

Use this command to install Jemalloc from the conda-forge channel. Jemalloc is a memory allocator that emphasizes fragmentation avoidance and scalable concurrency.

```bash
conda install -c conda-forge jemalloc
```