Intel Extension for PyTorch

https://intel.github.io/intel-extension-for-pytorch/
Admin
Intel Extension for PyTorch extends PyTorch with performance optimizations for Intel hardware,...
Tokens:343,654
Snippets:2,138
Trust Score:9.7
Update:6 days ago
Show doc for...
Tokens:
Raw
### Install documentation prerequisites

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/contribution.html

Install the documentation prerequisites from the docs directory. Run this before generating the HTML documentation.

```bash
cd docs
pip install -r requirements.txt

```

--------------------------------

### Basic CPU Installation

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/cheat_sheet.md.txt

Install Intel Extension for PyTorch for CPU using pip.

```bash
python -m pip install intel_extension_for_pytorch
```

--------------------------------

### Install documentation prerequisites

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Run these commands from the docs directory to install the required Python packages for building the documentation.

```bash
cd docs
pip install -r requirements.txt
```

--------------------------------

### Compilation Commands for C++ Example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/examples.md.txt

Commands to build the C++ example using CMake. Requires setting CMAKE_PREFIX_PATH to the libtorch path.

```bash
$ cd examples/cpu/inference/cpp
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=<LIBPYTORCH_PATH> ..
$ make
```

--------------------------------

### Replace Install with Develop Mode

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Replace 'python setup.py install' with 'python setup.py develop' to symlink Python files, avoiding reinstall for Python-only changes.

```bash
python setup.py install
```

```bash
python setup.py develop
```

--------------------------------

### Event log example using IPEX_EVENT_LOG and IPEX_INFO_EVENT_END

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/features/ipex_log.html

Demonstrates event logging for an operator calculation. Uses IPEX_EVENT_LOG to record start and intermediate steps, and IPEX_INFO_EVENT_END to complete the event. The event is identified by 'record_avg_pool'.

```cpp
IPEX_EVENT_LOG("OPS", "", "record_avg_pool", "start", "Here record the time start with arg:{}", arg);
prepare_data();
IPEX_EVENT_LOG("OPS", "", "record_avg_pool", "data_prepare_finish", "Here record the data_prepare_finish with arg:{}", arg);
avg_pool();
IPEX_INFO_EVENT_END("OPS", "", "record_avg_pool", "finish conv", "Here record the end");

```

--------------------------------

### Build documentation with pip and make

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/contribution.md.txt

Steps to build the documentation: first install the prerequisites from requirements.txt in the docs directory, then generate the HTML files into docs/_build/html.

```bash
cd docs
pip install -r requirements.txt
```

```bash
make clean
make html
```

--------------------------------

### Safely use USM allocations from Intel® Extension for PyTorch* via DLPack

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DLPack.md.txt

This example shows how to get a shared tensor from Intel® Extension for PyTorch* via DLPack and wait for the data to be ready using torch.xpu.synchronize() before using it. It is a possible solution for the consumer to safely use USM allocations.

```python
import intel_extension_for_pytorch
import torch.utils.dlpack

# Get shared tensor from Intel® Extension for PyTorch* via DLPack
t = torch.from_dlpack(capsule)

# Wait for the data ready to use
torch.xpu.synchronize()

# Use the data in shared tensor
...
```

--------------------------------

### Get data type size and Vec size with IPEX dispatch (GetVecLength)

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/features/isa_dynamic_dispatch.md.txt

These three files form a complete example of using IPEX's dispatch mechanism to get the data type size and vector size for a given at::ScalarType. The header declares the kernel function and dispatch stub, the .cpp defines the dispatch stub and calls it, and the kernel file implements the actual logic using at::vec::Vectorized<T>::size(). The kernel supports Double, Float, ComplexDouble, ComplexFloat, BFloat16, and Half, and throws a TORCH_CHECK error for unsupported types.

```c++
//csrc/cpu/aten/GetVecLength.h
#pragma once

#include <dyndisp/DispatchStub.h>

namespace torch_ipex {
namespace cpu {

std::tuple<int, int> get_cpp_typesize_and_vecsize(at::ScalarType dtype);

namespace {

std::tuple<int, int> get_cpp_typesize_and_vecsize_kernel_impl(
    at::ScalarType dtype);
}

using get_cpp_typesize_and_vecsize_kernel_fn =
    std::tuple<int, int> (*)(at::ScalarType);
IPEX_DECLARE_DISPATCH(
    get_cpp_typesize_and_vecsize_kernel_fn,
    get_cpp_typesize_and_vecsize_kernel_stub);

} // namespace cpu
} // namespace torch_ipex

```

```c++
//csrc/cpu/aten/GetVecLength.cpp

#include "GetVecLength.h"

namespace torch_ipex {
namespace cpu {

IPEX_DEFINE_DISPATCH(get_cpp_typesize_and_vecsize_kernel_stub);

// get cpp typesize and vectorsize by at::ScalarType
std::tuple<int, int> get_cpp_typesize_and_vecsize(at::ScalarType dtype) {
  return get_cpp_typesize_and_vecsize_kernel_stub(kCPU, dtype);
}

} // namespace cpu
} // namespace torch_ipex

```

```c++
//csrc/cpu/aten/kernels/GetVecLengthKrnl.cpp

#include <ATen/cpu/vec/vec.h>
#include "csrc/cpu/aten/GetVecLength.h"

namespace torch_ipex {
namespace cpu {

namespace {

std::tuple<int, int> get_cpp_typesize_and_vecsize_kernel_impl(
    at::ScalarType dtype) {
  switch (dtype) {
    case at::ScalarType::Double:
      return std::make_tuple(
          sizeof(double), at::vec::Vectorized<double>::size());
    case at::ScalarType::Float:
      return std::make_tuple(sizeof(float), at::vec::Vectorized<float>::size());
    case at::ScalarType::ComplexDouble:
      return std::make_tuple(
          sizeof(c10::complex<double>),
          at::vec::Vectorized<c10::complex<double>>::size());
    case at::ScalarType::ComplexFloat:
      return std::make_tuple(
          sizeof(c10::complex<float>),
          at::vec::Vectorized<c10::complex<float>>::size());
    case at::ScalarType::BFloat16:
      return std::make_tuple(
          sizeof(decltype(
              c10::impl::ScalarTypeToCPPType<at::ScalarType::BFloat16>::t)),
          at::vec::Vectorized<decltype(c10::impl::ScalarTypeToCPPType<
                                       at::ScalarType::BFloat16>::t)>::size());
    case at::ScalarType::Half:
      return std::make_tuple(
          sizeof(decltype(
              c10::impl::ScalarTypeToCPPType<at::ScalarType::Half>::t)),
          at::vec::Vectorized<decltype(c10::impl::ScalarTypeToCPPType<
                                       at::ScalarType::Half>::t)>::size());
    default:
      TORCH_CHECK(
          false,
          "Currently only floating and complex ScalarType are supported.");
  }
}

} // anonymous namespace

IPEX_REGISTER_DISPATCH(
    get_cpp_typesize_and_vecsize_kernel_stub,
    &get_cpp_typesize_and_vecsize_kernel_impl);

} // namespace cpu
} // namespace torch_ipex

```

--------------------------------

### Run Hypertune with example.yaml and resnet50.py

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/features/hypertune.html

Command to start Hypertune tuning for ncores_per_instance to minimize latency. Requires example.yaml and resnet50.py in the hypertune directory.

```bash
python -m intel_extension_for_pytorch.cpu.hypertune --conf_file <hypertune_directory>/example/example.yaml <hypertune_directory>/example/resnet50.py

```

--------------------------------

### ipex.optimize bfloat16 inference and training example

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/api_doc.html

Shows how to use ipex.optimize for bfloat16 inference and training. For inference, load the model state dict, set to eval mode, and call ipex.optimize with dtype=torch.bfloat16. For training, set model to train mode and pass the optimizer to get an optimized optimizer. Note: invoke optimize before DDP in distributed training.

```python
>>> # bfloat16 inference case.
>>> model = ...
>>> model.load_state_dict(torch.load(PATH))
>>> model.eval()
>>> optimized_model = ipex.optimize(model, dtype=torch.bfloat16)
>>> # running evaluation step.
>>> # bfloat16 training case.
>>> optimizer = ...
>>> model.train()
>>> optimized_model, optimized_optimizer = ipex.optimize(model, dtype=torch.bfloat16, optimizer=optimizer)
>>> # running training step.
```

--------------------------------

### setup.py for building LLTM DPC++ extension with setuptools

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DPC%2B%2B_Extension.md.txt

Use this setup.py script to build a DPC++ extension named 'lltm_xpu' from source files 'lltm_xpu.cpp' and 'lltm_xpu_kernel.cpp'. It uses DPCPPExtension and DpcppBuildExtension from torch.xpu.cpp_extension. Requires importing intel_extension_for_pytorch.

```python
from setuptools import setup
import torch
import intel_extension_for_pytorch
from torch.xpu.cpp_extension import DPCPPExtension, DpcppBuildExtension

setup(
    name='lltm',
    ext_modules=[
        DPCPPExtension('lltm_xpu', [
            'lltm_xpu.cpp',
            'lltm_xpu_kernel.cpp',
        ])
    ],
    cmdclass={
        'build_ext': DpcppBuildExtension
    })
```

--------------------------------

### Install Intel Optimization for Horovod

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/horovod.md.txt

Use pip to install the Intel Optimization for Horovod package. Requires oneAPI basekit to be installed.

```bash
python -m pip install intel-optimization-for-horovod
```

--------------------------------

### Install Intel Extension for PyTorch CPU wheel packages

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/releases.md.txt

Install the Intel Extension for PyTorch CPU wheel packages. The first command installs version 1.10.100 from the Intel software repository, while the second installs the latest version from PyPI.

```python
python -m pip install intel_extension_for_pytorch==1.10.100 -f https://software.intel.com/ipex-whl-stable
```

```python
pip install intel_extension_for_pytorch
```

--------------------------------

### Simple log example using IPEX_INFO_LOG

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/features/ipex_log.html

Shows how to use the simple log macro IPEX_INFO_LOG with a log component, sub-component, format string, and argument. The example logs inside an 'abs' kernel.

```cpp
IPEX_INFO_LOG("OPS", "", "Add a log for inside ops {}", "abs");

```

--------------------------------

### Install Intel Extension for PyTorch 1.9.0

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/releases.md.txt

Install the Intel Extension for PyTorch 1.9.0 binary package from the Intel software repository. Use this command to install the extension via pip.

```bash
python -m pip install torch_ipex==1.9.0 -f https://software.intel.com/ipex-whl-stable
```

--------------------------------

### Vanilla setuptools Extension equivalent for LLTM

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/_sources/tutorials/features/DPC%2B%2B_Extension.md.txt

This is the equivalent vanilla setuptools Extension configuration for the DPC++ extension, showing the include paths and language setting. It is provided as a comparison to the DPCPPExtension wrapper.

```python
Extension(
   name='lltm_xpu',
   sources=['lltm_xpu.cpp', 'lltm_xpu_kernel.cpp',],
   include_dirs=cpp_extension.include_paths(),
   language='c++')
```

--------------------------------

### Install linting dependencies

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Install extra dependencies required for linting. This command should only need to be run once.

```bash
make setup_lint
```

--------------------------------

### Run a specific instance with --instance-idx 1

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/launch_script.md.txt

Launches only instance 1 of 4 instances. The log file shows the environment and command used for that instance.

```bash
ipexrun --ninstances 4 --instance-idx 1 --log-dir ./logs resnet50.py
```

--------------------------------

### Install CMake with Homebrew

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Recommended way to install CMake on macOS or Linux using Homebrew.

```bash
brew install cmake
```

--------------------------------

### View instance log for all physical cores

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/launch_script.md.txt

Displays the contents of the instances log file, showing environment variables and the command used for the launch.

```bash
$ cat logs/run_20210712212258_instances.log
2021-07-12 21:22:58,764 - __main__ - WARNING - Both TCMalloc and JeMalloc are not found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /home/<user>/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance
2021-07-12 21:22:58,764 - __main__ - INFO - OMP_NUM_THREADS=44
2021-07-12 21:22:58,764 - __main__ - INFO - Using Intel OpenMP
2021-07-12 21:22:58,764 - __main__ - INFO - KMP_AFFINITY=granularity=fine,compact,1,0
2021-07-12 21:22:58,764 - __main__ - INFO - KMP_BLOCKTIME=1
2021-07-12 21:22:58,764 - __main__ - INFO - LD_PRELOAD=<VIRTUAL_ENV>/lib/libiomp5.so
2021-07-12 21:22:58,764 - __main__ - WARNING - Numa Aware: cores:['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43'] on different NUMA nodes
2021-07-12 21:22:58,764 - __main__ - INFO - numactl -C 0-43 <VIRTUAL_ENV>/bin/python resnet50.py 2>&1 | tee ./logs/run_20210712212258_instance_0_cores_0-43.log
```

--------------------------------

### Install pytest

Source: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/contribution.html

Install pytest via pip. It is not officially supported but works well with the unit tests.

```bash
pip install pytest
```

--------------------------------

### Run linting with make

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/contribution.md.txt

Run the same linting steps used in CI locally. Use 'make lint' for all files or 'make quicklint' for changed files, with parallel jobs.

```bash
# Lint all files
make lint -j 6  # run lint (using 6 parallel jobs)

# Lint only the files you have changed
make quicklint -j 6
```

--------------------------------

### View instance log for all cores including logical cores

Source: https://intel.github.io/intel-extension-for-pytorch/cpu/latest/_sources/tutorials/performance_tuning/launch_script.md.txt

Displays the contents of the instances log file, showing environment variables and the command used for the launch.

```bash
$ cat logs/run_20210712223308_instances.log
2021-07-12 22:33:08,117 - __main__ - WARNING - Both TCMalloc and JeMalloc are not found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /home/<user>/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance
2021-07-12 22:33:08,117 - __main__ - INFO - OMP_NUM_THREADS=88
2021-07-12 22:33:08,117 - __main__ - INFO - Using Intel OpenMP
2021-07-12 22:33:08,118 - __main__ - INFO - KMP_AFFINITY=granularity=fine,compact,1,0
2021-07-12 22:33:08,118 - __main__ - INFO - KMP_BLOCKTIME=1
2021-07-12 22:33:08,118 - __main__ - INFO - LD_PRELOAD=<VIRTUAL_ENV>/lib/libiomp5.so
```
Tokens:
Raw