### Getting Started with PyTorch

Source: https://pytorch.ac.cn/docs/2.4/mtia

Learn how to get started with PyTorch by running it locally or using a supported cloud platform. This section provides initial steps for new users.

```APIDOC
## Getting Started

### Description
Quickly get started with PyTorch by running it locally or using one of the supported cloud platforms.

### Resources
*   **Local Setup**: Instructions for installing and running PyTorch on your machine.
*   **Cloud Platforms**: Information on using PyTorch with various cloud services.
```

--------------------------------

### Example: Starting Two '/usr/bin/touch' Binary Copies

Source: https://pytorch.ac.cn/docs/2.4/elastic/multiprocessing

Shows how to use `start_processes` to launch two instances of the '/usr/bin/touch' command with different file arguments. This example confirms that external binaries can be executed, and their arguments are passed correctly.

```python
# ok; two copies of /usr/bin/touch: touch file1, touch file2
start_processes(
   name="trainer",
   entrypoint="/usr/bin/touch",
   args:{0:("file1",), 1:("file2",),
   envs:{0:{}, 1:{}},
   log_dir=log_dir
 )

```

--------------------------------

### Learning Resources

Source: https://pytorch.ac.cn/docs/2.4/cpu

Explore various learning materials to get started with PyTorch, including tutorials, recipes, and introductory videos.

```APIDOC
## Learning

### Get Started
Run PyTorch locally or quickly get started with one of the supported cloud platforms.

### Tutorials
What's new in PyTorch tutorials.

### Learn the Basics
Familiarize yourself with PyTorch concepts and modules.

### PyTorch Recipes
Concise and easy-to-use PyTorch code examples that are ready to deploy.

### Introduction to PyTorch - YouTube Series
Master PyTorch fundamentals with our engaging YouTube tutorial series.
```

--------------------------------

### Example: Starting Two 'foo' Function Copies

Source: https://pytorch.ac.cn/docs/2.4/elastic/multiprocessing

Illustrates a valid use case for `start_processes` where two copies of a function 'foo' are started with distinct arguments. This example highlights the requirement for `args` and `envs` to have matching key sets for all local ranks.

```python
log_dir = "/tmp/test"

# ok; two copies of foo: foo("bar0"), foo("bar1")
start_processes(
   name="trainer",
   entrypoint=foo,
   args:{0:("bar0",), 1:("bar1",),
   envs:{0:{}, 1:{}},
   log_dir=log_dir
)

```

--------------------------------

### Inference Example with FP32 on Intel GPU

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

This Python code demonstrates an inference workflow using FP32 precision on an Intel GPU. It loads a ResNet50 model, moves the model and input data to the 'xpu' device, and performs inference. This example highlights the straightforward device placement for Intel GPUs.

```python
import torch
import torchvision.models as models

model = models.resnet50(weights="ResNet50_Weights.DEFAULT")
model.eval()
data = torch.rand(1, 3, 224, 224)

######## code changes #######
model = model.to("xpu")
data = data.to("xpu")
######## code changes #######

with torch.no_grad():
    model(data)

print("Execution finished")
```

--------------------------------

### Build PyTorch from Source for Intel GPUs

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

This comprehensive snippet details the process of building PyTorch, torchvision, and torchaudio from their source code for Intel GPUs. It includes cloning repositories, checking out specific versions, installing dependencies, setting environment variables like USE_XPU=1, and using make commands for compilation. This process is essential for enabling Intel GPU support.

```bash
# Get PyTorch Source Code
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
git checkout main # or checkout the specific release version >= v2.4
git submodule sync
git submodule update --init --recursive

# Get required packages for compilation
conda install cmake ninja
pip install -r requirements.txt

# Pytorch for Intel GPUs only support Linux platform for now.
# Install the required packages for pytorch compilation.
conda install intel::mkl-static intel::mkl-include

# (optional) If using torch.compile with inductor/triton, install the matching version of triton
# Run from the pytorch directory after cloning
# For Intel GPU support, please explicitly `export USE_XPU=1` before running command.
USE_XPU=1 make triton

# If you would like to compile PyTorch with new C++ ABI enabled, then first run this command:
export _GLIBCXX_USE_CXX11_ABI=1

# pytorch build from source
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py develop
cd ..

# (optional) If using torchvison.
# Get torchvision Code
git clone https://github.com/pytorch/vision.git
cd vision
python setup.py develop
cd ..

# (optional) If using torchaudio.
# Get torchaudio Code
git clone https://github.com/pytorch/audio.git
cd audio
pip install -r requirements.txt
git checkout main # or specific version
git submodule sync
git submodule update --init --recursive
python setup.py develop
cd ..
```

--------------------------------

### 使用 FP32 进行训练（Python）

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

此代码片段展示了使用 FP32 精度在 Intel GPU 上训练 ResNet50 模型。它设置了 CIFAR-10 数据集、模型、损失函数和优化器，然后将它们移动到 'xpu' 设备。训练循环包括前向和后向传递以及优化器步骤。

```python
import torch
import torchvision

LR = 0.001
DOWNLOAD = True
DATA = "datasets/cifar10/"

transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize((224, 224)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    ]
)
train_dataset = torchvision.datasets.CIFAR10(
    root=DATA,
    train=True,
    transform=transform,
    download=DOWNLOAD,
)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=128)

model = torchvision.models.resnet50()
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=LR, momentum=0.9)
model.train()
######################## code changes #######################
model = model.to("xpu")
criterion = criterion.to("xpu")
######################## code changes #######################

for batch_idx, (data, target) in enumerate(train_loader):
    ########## code changes ##########
    data = data.to("xpu")
    target = target.to("xpu")
    ########## code changes ##########
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print(batch_idx)
torch.save(
    {
        "model_state_dict": model.state_dict(),
        "optimizer_state_dict": optimizer.state_dict(),
    },
    "checkpoint.pth",
)

print("Execution finished")

```

--------------------------------

### 使用 AMP 进行训练（Python）

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

此代码片段演示了如何使用自动混合精度（AMP）在 Intel GPU 上训练 ResNet50 模型。它集成了 `torch.amp.GradScaler` 和 `torch.autocast` 来在 FP16 和 FP32 之间动态切换，以提高训练速度和效率。

```python
import torch
import torchvision

LR = 0.001
DOWNLOAD = True
DATA = "datasets/cifar10/"

use_amp=True

transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize((224, 224)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    ]
)
train_dataset = torchvision.datasets.CIFAR10(
    root=DATA,
    train=True,
    transform=transform,
    download=DOWNLOAD,
)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=128)

model = torchvision.models.resnet50()
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=LR, momentum=0.9)
scaler = torch.amp.GradScaler(enabled=use_amp)

model.train()
######################## code changes #######################
model = model.to("xpu")
criterion = criterion.to("xpu")
######################## code changes #######################

for batch_idx, (data, target) in enumerate(train_loader):
    ########## code changes ##########
    data = data.to("xpu")
    target = target.to("xpu")
    ########## code changes ##########
    # set dtype=torch.bfloat16 for BF16
    with torch.autocast(device_type="xpu", dtype=torch.float16, enabled=use_amp):
        output = model(data)
        loss = criterion(output, target)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()
    print(batch_idx)

torch.save(
    {
        "model_state_dict": model.state_dict(),
        "optimizer_state_dict": optimizer.state_dict(),
    },
    "checkpoint.pth",
)

print("Execution finished")

```

--------------------------------

### 使用 torch.compile 进行推理（Python）

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

此代码片段演示如何使用 `torch.compile` 优化 PyTorch 模型以进行推理。它将模型和数据移动到 'xpu' 设备，然后编译模型以提高性能。最后，它执行指定次数的推理。

```python
import torch
import torchvision.models as models

model = models.resnet50(weights="ResNet50_Weights.DEFAULT")
model.eval()
data = torch.rand(1, 3, 224, 224)
ITERS = 10

######## code changes #######
model = model.to("xpu")
data = data.to("xpu")
######## code changes #######

model = torch.compile(model)
for i in range(ITERS):
    with torch.no_grad():
        model(data)

print("Execution finished")

```

--------------------------------

### Install and Run TensorBoard

Source: https://pytorch.ac.cn/docs/2.4/tensorboard

These commands show how to install the TensorBoard package using pip and how to launch the TensorBoard server, specifying the directory where PyTorch logs are saved. This is essential for visualizing the logged data.

```bash
pip install tensorboard
tensorboard --logdir=runs
```

--------------------------------

### Start Multiple Echo Workers as Binaries with Redirects

Source: https://pytorch.ac.cn/docs/2.4/elastic/multiprocessing

Launches multiple instances of an external binary (e.g., 'echo') as worker processes using `start_processes`. This example demonstrates how to pass arguments to the binary and redirect its standard output to a log file for a specific rank. It's useful for running external command-line tools in a distributed manner.

```python
# same as invoking
# echo hello
# echo world > stdout.log
ctx = start_processes(
        name="echo"
        entrypoint="echo",
        log_dir="/tmp/foobar",
        args={0: "hello", 1: "world"},
        redirects={1: Std.OUT},
       )

```

--------------------------------

### Setup for Diagnosing Backend Compiler Errors (General)

Source: https://pytorch.ac.cn/docs/2.4/torch.compiler_troubleshooting

This Python code snippet initializes a PyTorch model and imports necessary modules from `torch._dynamo`. It serves as a starting point for diagnosing errors in various backend compilers, not just TorchInductor. The setup is similar to the TorchInductor specific examples but is intended for broader application when using different backends.

```python
import torch

import torch._dynamo as dynamo

model = torch.nn.Sequential(*[torch.nn.Linear(200, 200) for _ in range(5)])

```

--------------------------------

### Initialize and Use FileStore for Persistent Key-Value Storage (Python)

Source: https://pytorch.ac.cn/docs/2.4/distributed

Illustrates the setup and usage of a FileStore, which persists key-value pairs to a file. This allows data to survive process restarts and can be accessed by multiple processes by referencing the same file path.

```Python
import torch.distributed as dist

store1 = dist.FileStore("/tmp/filestore", 2)
store2 = dist.FileStore("/tmp/filestore", 2)

# Use any of the store methods from either the client or server after initialization
store1.set("first_key", "first_value")
store2.get("first_key")

```

--------------------------------

### Set Up Environment for Intel GPU with PyTorch

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

This snippet shows how to set up the environment for PyTorch development on Intel GPUs by sourcing the setvars.sh script. It requires the ONEAPI_ROOT environment variable to be set, pointing to the installation directory of intel-for-pytorch-gpu-dev and intel-pti-dev packages.

```bash
source ${ONEAPI_ROOT}/setvars.sh
```

--------------------------------

### Get Help for torch.distributed.launch (Python)

Source: https://pytorch.ac.cn/docs/2.4/distributed

This command displays all available optional arguments and their descriptions for the torch.distributed.launch utility. It is useful for understanding the full range of configuration options.

```python
python -m torch.distributed.launch --help
```

--------------------------------

### Check Intel GPU Availability with PyTorch

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

This Python snippet demonstrates how to check if an Intel GPU is available and recognized by PyTorch. It uses the `torch.xpu.is_available()` function, which is specific to Intel GPU support. If it returns `False`, it suggests verifying the GPU installation and PyTorch compilation.

```python
import torch
torch.xpu.is_available()  # torch.xpu is the API for Intel GPU support
```

--------------------------------

### Inference Example with AMP on Intel GPU

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

This Python snippet shows how to perform inference using Automatic Mixed Precision (AMP) on an Intel GPU. It configures the model and data to use the 'xpu' device and enables AMP with `torch.autocast`, specifying `dtype=torch.float16` for mixed precision.

```python
import torch
import torchvision.models as models

model = models.resnet50(weights="ResNet50_Weights.DEFAULT")
model.eval()
data = torch.rand(1, 3, 224, 224)

#################### code changes #################
model = model.to("xpu")
data = data.to("xpu")
#################### code changes #################

with torch.no_grad():
    d = torch.rand(1, 3, 224, 224)
    ############################# code changes #####################
    d = d.to("xpu")
    # set dtype=torch.bfloat16 for BF16
    with torch.autocast(device_type="xpu", dtype=torch.float16, enabled=True):
    ############################# code changes #####################
        model(data)

print("Execution finished")
```

--------------------------------

### Initializing RPC with TensorPipe Backend

Source: https://pytorch.ac.cn/docs/2.4/rpc

Example of initializing the RPC system with the TensorPipe backend, including setting up backend options.

```APIDOC
## Initialize RPC with TensorPipe

### Description
This example demonstrates how to initialize the distributed RPC environment using the TensorPipe backend. It includes setting environment variables, creating `TensorPipeRpcBackendOptions`, and calling `rpc.init_rpc`.

### Method
`rpc.init_rpc`

### Endpoint
N/A (Initialization function)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
import os
import torch
from torch.distributed import rpc

# Set environment variables for distributed setup
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'

# Define TensorPipe backend options
options = rpc.TensorPipeRpcBackendOptions(
    num_worker_threads=8,
    rpc_timeout=20 # 20 second timeout
    # Example of setting device maps directly in options
    # device_maps={\"worker1\": {0: 1}} 
)

# Optionally, set device maps using the method
# options.set_device_map("worker1", {1: 2})

# Initialize RPC on worker0
try:
    rpc.init_rpc(
        "worker0",
        rank=0,
        world_size=2,
        backend=rpc.BackendType.TENSORPIPE,
        rpc_backend_options=options
    )
    print("Worker0 initialized successfully.")
except Exception as e:
    print(f"Error initializing Worker0: {e}")

# In a real distributed setting, worker1 would also initialize RPC:
# rpc.init_rpc(
#     "worker1",
#     rank=1,
#     world_size=2,
#     backend=rpc.BackendType.TENSORPIPE,
#     rpc_backend_options=options
# )

# Example of a remote call (assuming worker1 is also initialized)
# def add(x, y):
#     return x + y
# 
# if rpc.is_initialized():
#     x = torch.ones(2).to(0) # Send tensor on device 0
#     try:
#         # The tensor will be moved to worker1's device 1 due to device_maps
#         rets = rpc.rpc_sync("worker1", add, args=(x, 1))
#         print(f"RPC sync result: {rets}")
#     except Exception as e:
#         print(f"RPC sync failed: {e}")

# Clean up RPC
# if rpc.is_initialized():
#     rpc.shutdown()
```

### Response
#### Success Response (200)
Indicates successful initialization of the RPC agent. No specific return value, but subsequent RPC calls will succeed.

#### Response Example
```
Worker0 initialized successfully.
```
```

--------------------------------

### Example Graph with Get Attr Node

Source: https://pytorch.ac.cn/docs/2.4/export.ir_spec

Presents an example FX graph that includes a 'get_attr' node. This specific example shows how submodules like 'true_graph_0' and 'false_graph_0' are accessed.

```Python
graph():
    %x_1 : [num_users=1] = placeholder[target=x_1]
    %y_1 : [num_users=1] = placeholder[target=y_1]
    %true_graph_0 : [num_users=1] = get_attr[target=true_graph_0]
    %false_graph_0 : [num_users=1] = get_attr[target=false_graph_0]
    %conditional : [num_users=1] = call_function[target=torch.ops.higher_order.cond](args = (%y_1, %true_graph_0, %false_graph_0, [%x_1]), kwargs = {})
    return conditional

```

--------------------------------

### FileStore Initialization and Usage

Source: https://pytorch.ac.cn/docs/2.4/distributed

Details the FileStore, which uses a file to persist key-value pairs, and provides an example of its initialization and usage across multiple processes.

```APIDOC
## FileStore

### Description
Store implementation that uses a file to store key-value pairs.

### Method
`torch.distributed.FileStore(file_name: str, world_size: Optional[int] = -1)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
import torch.distributed as dist

store1 = dist.FileStore("/tmp/filestore", 2)
store2 = dist.FileStore("/tmp/filestore", 2)

# Use any of the store methods from either the client or server after initialization
store1.set("first_key", "first_value")
store2.get("first_key")
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Migrate CUDA Code to Intel GPU (XPU)

Source: https://pytorch.ac.cn/docs/2.4/notes/get_start_xpu

This example illustrates the minimal code change required when migrating from CUDA to Intel GPU support in PyTorch. It shows how to replace `.to("cuda")` with `.to("xpu")` for tensors to utilize the Intel GPU.

```python
# CUDA CODE
tensor = torch.tensor([1.0, 2.0]).to("cuda")

# CODE for Intel GPU
tensor = torch.tensor([1.0, 2.0]).to("xpu")
```

--------------------------------

### TCPStore Initialization and Usage

Source: https://pytorch.ac.cn/docs/2.4/distributed

Demonstrates how to initialize and use the TCPStore for distributed communication. It covers setting up a server store and a client store, and performing basic operations like set and get.

```APIDOC
## TCPStore

### Description
Based on TCP, this distributed key-value store implementation uses a server to store data, while clients connect via TCP to perform operations like `set()` and `get()`.

### Method
`torch.distributed.TCPStore(host_name: str, port: int, world_size: Optional[int] = None, is_master: bool = False, timeout: timedelta = timedelta(seconds=300), wait_for_workers: bool = True, multi_tenant: bool = False, master_listen_fd: Optional[int] = None, use_libuv: bool = True)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
import torch.distributed as dist
from datetime import timedelta

# Run on process 1 (server)
server_store = dist.TCPStore("127.0.0.1", 1234, 2, True, timedelta(seconds=30))

# Run on process 2 (client)
client_store = dist.TCPStore("127.0.0.1", 1234, 2, False)

# Use any of the store methods from either the client or server after initialization
server_store.set("first_key", "first_value")
client_store.get("first_key")
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Initialize and Use TCPStore for Distributed Communication (Python)

Source: https://pytorch.ac.cn/docs/2.4/distributed

Demonstrates setting up a TCPStore for distributed communication. It involves initializing a server store and a client store, then using them to set and get key-value pairs. This is useful for sharing data between processes in a distributed PyTorch environment.

```Python
import torch.distributed as dist
from datetime import timedelta

# Run on process 1 (server)
server_store = dist.TCPStore("127.0.0.1", 1234, 2, True, timedelta(seconds=30))
# Run on process 2 (client)
client_store = dist.TCPStore("127.0.0.1", 1234, 2, False)

# Use any of the store methods from either the client or server after initialization
server_store.set("first_key", "first_value")
client_store.get("first_key")

```

--------------------------------

### PyTorch Tutorials

Source: https://pytorch.ac.cn/docs/2.4/mtia

Explore a collection of tutorials designed to help you learn PyTorch, from fundamental concepts to practical applications.

```APIDOC
## Tutorials

### Description
Discover new content in PyTorch tutorials. Learn the basics, familiarize yourself with PyTorch concepts and modules, and find short, deployable PyTorch code examples.

### Content
*   **What's New**: Latest additions to the PyTorch tutorial series.
*   **Learn the Basics**: Understand fundamental PyTorch concepts.
*   **PyTorch Tips**: Concise code examples for immediate use.
*   **Introduction to PyTorch (YouTube Series)**: Master PyTorch fundamentals through engaging video tutorials.
```

--------------------------------

### HashStore Initialization and Usage

Source: https://pytorch.ac.cn/docs/2.4/distributed

Explains how to use the HashStore, a thread-safe implementation based on an underlying hash map, suitable for use within a single process.

```APIDOC
## HashStore

### Description
Thread-safe store implementation based on an underlying hash map. This store can be used within the same process (e.g., by other threads), but not across processes.

### Method
`torch.distributed.HashStore()`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
import torch.distributed as dist

store = dist.HashStore()

# store can be used from other threads
# Use any of the store methods after initialization
store.set("first_key", "first_value")
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Launch Multi-Node Distributed Training (Node 1) (Python)

Source: https://pytorch.ac.cn/docs/2.4/distributed

This command initiates a multi-node distributed training session from the first node. It requires specifying the total number of nodes, the rank of the current node, the master node's address, and the master port. The number of processes per node and the training script are also provided.

```python
python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE --nnodes=2 --node_rank=0 --master_addr="192.168.1.1" --master_port=1234 YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3 and all other arguments of your training script)
```

--------------------------------

### Distributed Backward Pass with RRef

Source: https://pytorch.ac.cn/docs/2.4/rpc

This example demonstrates how to perform a distributed backward pass starting from an RRef. It uses `torch.distributed.autograd.context` to manage the distributed autograd context and calls the `backward()` method on the RRef.

```python
import torch.distributed.autograd as dist_autograd
with dist_autograd.context() as context_id:
    rref.backward(context_id)
```

--------------------------------

### Launch Multi-Node Distributed Training (Node 2) (Python)

Source: https://pytorch.ac.cn/docs/2.4/distributed

This command is executed on the second node for a multi-node distributed training setup. It mirrors the configuration of the first node, specifying the total number of nodes, its own rank, the master node's address, and port. The number of processes per node and the training script are also included.

```python
python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE --nnodes=2 --node_rank=1 --master_addr="192.168.1.1" --master_port=1234 YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3 and all other arguments of your training script)
```

--------------------------------

### Initialize Distributed Training and Debugging

Source: https://pytorch.ac.cn/docs/2.4/distributed

This snippet demonstrates initializing a distributed training environment by setting master address and port, enabling detailed distributed debugging, and launching worker processes. It's crucial for debugging collective operation synchronization issues.

```python
import os
import torch.distributed as dist
import torch.multiprocessing as mp

def worker(rank):
    # ... worker logic ...
    tensor = torch.randn(10 if rank == 0 else 20).cuda()
    dist.all_reduce(tensor)
    torch.cuda.synchronize(device=rank)

if __name__ == "__main__":
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "29501"
    os.environ["TORCH_CPP_LOG_LEVEL"]="INFO"
    os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"
    mp.spawn(worker, nprocs=2, args=())
```

--------------------------------

### Get Ignored Functions in PyTorch Overrides

Source: https://pytorch.ac.cn/docs/2.4/torch.overrides

Retrieves a set of public functions in the PyTorch API that cannot be overridden by `__torch_function__`. This is typically because their arguments are not tensors or tensor-like. The example demonstrates checking if `torch.Tensor.as_subclass` is in the ignored set and if `torch.add` is not.

```python
>>> import torch
>>> torch.Tensor.as_subclass in torch.overrides.get_ignored_functions()
True
>>> torch.add in torch.overrides.get_ignored_functions()
False
```

--------------------------------

### Set Default Timeout in PyTorch Distributed Store

Source: https://pytorch.ac.cn/docs/2.4/distributed

Demonstrates how to change the default timeout for the distributed store using `set_timeout`. This new timeout will be used in subsequent `wait` and `get` operations, as well as the initial store setup. This allows for dynamic adjustment of synchronization periods.

```python
import torch.distributed as dist
from datetime import timedelta
# Using TCPStore as an example, other store types can also be used
store = dist.TCPStore("127.0.0.1", 0, 1, True, timedelta(seconds=30))
store.set_timeout(timedelta(seconds=10))
# This will throw an exception after 10 seconds
store.wait(["bad_key"])
```

--------------------------------

### PyTorch Tutorials

Source: https://pytorch.ac.cn/docs/2.4/fft

Offers in-depth tutorials for both beginner and advanced PyTorch developers.

```APIDOC
## Tutorials

### Description
Get in-depth tutorials for beginner and advanced developers.

### Endpoint
`/tutorials`

### Method
GET

### Parameters
None

### Response
- **Tutorial Content** (HTML/Markdown) - Step-by-step guides and examples for various PyTorch functionalities.
```

--------------------------------

### Get Testing Overrides in PyTorch

Source: https://pytorch.ac.cn/docs/2.4/torch.overrides

Provides a dictionary of dummy overrides for all overridable functions, useful for testing `__torch_function__` implementations. Each override is a lambda function that returns -1, regardless of the input. The example uses `inspect.signature` to show the signature of the dummy override for `torch.add`.

```python
>>> import inspect
>>> import torch
>>> my_add = torch.overrides.get_testing_overrides()[torch.add]
>>> inspect.signature(my_add)
<Signature (input, other, out=None)>
```

--------------------------------

### Resolve Function Name in PyTorch Overrides

Source: https://pytorch.ac.cn/docs/2.4/torch.overrides

Obtains a human-readable string name for a function passed to `__torch_function__`. This utility is useful for debugging and logging purposes, providing a clear identifier for the function being processed. The example shows how to get the name of a given function `f`.

```python
# Example usage (conceptual):
# def my_func(): pass
# func_name = torch.overrides.resolve_name(my_func)
# print(func_name)
```

--------------------------------

### RemoteModule Initialization and Usage

Source: https://pytorch.ac.cn/docs/2.4/rpc

Illustrates how to create and use a `RemoteModule` for running `nn.Module` instances on remote devices. It shows asynchronous and synchronous forward calls, handling of RPC initialization, and specifies the format for `remote_device`.

```python
import torch
import torch.distributed.rpc as rpc
from torch import nn, Tensor
from torch.distributed.nn.api.remote_module import RemoteModule

rpc.init_rpc("worker0", rank=0, world_size=2)
remote_linear_module = RemoteModule(
    "worker1/cpu", nn.Linear, args=(20, 30),
)
input = torch.randn(128, 20)
ret_fut = remote_linear_module.forward_async(input)
ret = ret_fut.wait()
rpc.shutdown()
```

```python
import torch
import torch.distributed.rpc as rpc

rpc.init_rpc("worker1", rank=1, world_size=2)
rpc.shutdown()
```

--------------------------------

### Sharing CUDA Tensors Between Processes in PyTorch

Source: https://pytorch.ac.cn/docs/2.4/multiprocessing

Guidelines and examples for sharing CUDA tensors between processes using `torch.multiprocessing`. This functionality is supported in Python 3 with 'spawn' or 'forkserver' start methods. It emphasizes best practices to prevent memory leaks and ensure proper resource management.

```python
# Example of good practice for consuming shared CUDA tensors
import torch
import torch.multiprocessing as mp

# Assume 'queue' is a multiprocessing Queue populated with CUDA tensors
# queue = mp.Queue()

def consumer_process(queue):
    x = queue.get() # Receive the shared tensor
    # ... do something with x ...
    del x # Release memory as soon as possible

# Example of bad practice (holding onto tensor longer than needed)
# def bad_consumer_process(queue):
#     x = queue.get()
#     # ... do something with x ...
#     # Memory is not released here, potentially causing issues

# Example of passing a received tensor (not recommended)
# def passing_received_tensor(queue, queue_2):
#     x = queue.get()
#     # This is problematic as it might lead to issues if not handled carefully
#     # queue_2.put(x)

# Example of creating a process-local copy
def safe_passing_tensor(queue, queue_2):
    x = queue.get()
    x_clone = x.clone() # Create a deep copy
    queue_2.put(x_clone)

# Example of putting and getting from the same queue in the same process (likely segfault)
# def same_process_queue_issue(queue, tensor):
#     queue.put(tensor)
#     x = queue.get() # This can lead to a segfault

```

--------------------------------

### File System Initialization

Source: https://pytorch.ac.cn/docs/2.4/distributed

Initialize a distributed process group using a shared file system. The URL should start with `file://` and point to a file that will be created by the initialization process. Ensure the file is cleaned up after use to prevent deadlocks.

```APIDOC
## File System Initialization

### Description
Initialize a distributed process group using a shared file system. The URL should start with `file://` and point to a file that will be created by the initialization process. Ensure the file is cleaned up after use to prevent deadlocks.

### Method
`torch.distributed.init_process_group`

### Endpoint
`file://<path_to_shared_file>`

### Parameters
#### Path Parameters
- **path_to_shared_file** (str) - Required - A path on the shared file system that does not exist. The initialization process will create this file.

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
import torch.distributed as dist

# rank should always be specified
dist.init_process_group(backend, init_method='file:///mnt/nfs/sharedfile', world_size=4, rank=args.rank)
```

### Response
#### Success Response (200)
Initializes the process group.

#### Response Example
None (in-place initialization)
```

--------------------------------

### Single-Node Multi-Worker Training with torchrun

Source: https://pytorch.ac.cn/docs/2.4/elastic/run

This example demonstrates how to configure `torchrun` for single-node multi-worker distributed training. It specifies the number of nodes (1), the number of processes per node, and points to the training script. The `--standalone` flag indicates a single-node setup.

```bash
torchrun \
    --standalone \
    --nnodes=1 \
    --nproc-per-node=$NUM_TRAINERS \
    YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)
```

--------------------------------

### Quantization Aware Training (QAT) with x86 Backend

Source: https://pytorch.ac.cn/docs/2.4/quantization

This snippet demonstrates setting up a model for Quantization Aware Training (QAT) using the 'x86' backend. It involves fusing modules, preparing the model with observers, running a training loop (not shown), and finally converting the observed model to an 8-bit quantized model.

```python
import torch

# Assume model_fp32 and input_fp32 are defined elsewhere

# Set QAT config for x86 backend
model_fp32.qconfig = torch.ao.quantization.get_default_qat_qconfig('x86')

# Fuse modules (e.g., conv, bn, relu)
model_fp32_fused = torch.ao.quantization.fuse_modules(model_fp32,
    [['conv', 'bn', 'relu']])

# Prepare the model for QAT by inserting observers
model_fp32_prepared = torch.ao.quantization.prepare_qat(model_fp32_fused.train())

# Placeholder for the training loop
# training_loop(model_fp32_prepared)

# Convert the observed model to an 8-bit quantized model
model_fp32_prepared.eval()
model_int8 = torch.ao.quantization.convert(model_fp32_prepared)

# Run the quantized model
res = model_int8(input_fp32)
```

--------------------------------

### CUDA Graphs with DistributedDataParallel (DDP) - NCCL >= 2.9.6

Source: https://pytorch.ac.cn/docs/2.4/notes/cuda

This example demonstrates capturing the entire backward pass with DDP when using NCCL version 2.9.6 or later. It requires three setup steps: disabling NCCL's internal asynchronous error handling, building DDP within a side stream context, and ensuring sufficient warmup iterations (at least 11) in eager mode before capturing.

```python
# 1. Disable DDP's internal asynchronous error handling
os.environ["NCCL_ASYNC_ERROR_HANDLING"] = "0"
torch.distributed.init_process_group(...)

# 2. Build DDP in a side stream context before full backward capture
s = torch.cuda.Stream()
with torch.cuda.stream(s):
    model = DistributedDataParallel(model)

# 3. Warmup must run at least 11 DDP-enabled eager mode iterations before capture.
```

--------------------------------

### Install and Use depyf for Bytecode Decompilation

Source: https://pytorch.ac.cn/docs/2.4/torch.compiler_dynamo_overview

Install the depyf library to decompile Python bytecode into human-readable source code. This is useful for understanding the compiled code generated by Dynamo. Ensure depyf is installed via pip before using it.

```python
import depyf
depyf.install()

```