### SyclExtension Example

Source: https://docs.pytorch.org/docs/2.12/cpp_extension.html

Example of using SyclExtension to create a setuptools.Extension for SYCL/C++ extensions, including build setup.

```APIDOC
## SyclExtension Example

### Description
This example shows how to use `SyclExtension` to create a `setuptools.Extension` for SYCL/C++ extensions. It demonstrates setting up the build process with PyTorch's `BuildExtension` and specifying compilation flags. It also highlights the warning regarding `py_limited_api` and how to manage device architecture targeting with `TORCH_XPU_ARCH_LIST`.

### Usage
```python
from torch.utils.cpp_extension import BuildExtension, SyclExtension
from setuptools import setup

setup(
    name='xpu_extension',
    ext_modules=[
    SyclExtension(
                name='xpu_extension',
                sources=['extension.cpp', 'extension_kernel.cpp'],
                extra_compile_args={'cxx': ['-g', '-std=c++20', '-fPIC']})
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
```

### Parameters
- `name` (str): The name of the extension.
- `sources` (list[str]): A list of source file paths.
- `extra_compile_args` (dict): A dictionary containing extra compile arguments for 'cxx'.

### Environment Variable
- `TORCH_XPU_ARCH_LIST` (str): Specifies the device architectures to support (e.g., `"pvc,xe-lpg"`).
```

--------------------------------

### Initialize and Use TCPStore

Source: https://docs.pytorch.org/docs/2.12/distributed.html

Demonstrates initializing TCPStore on a server and client process, and then setting and getting a key-value pair. Ensure the server is started before the client attempts to connect.

```python
import torch.distributed as dist
from datetime import timedelta
# Run on process 1 (server)
server_store = dist.TCPStore("127.0.0.1", 1234, 2, True, timedelta(seconds=30))
# Run on process 2 (client)
client_store = dist.TCPStore("127.0.0.1", 1234, 2, False)
# Use any of the store methods from either the client or server after initialization
server_store.set("first_key", "first_value")
client_store.get("first_key")
```

--------------------------------

### Usage Examples

Source: https://docs.pytorch.org/docs/2.12/_sources/backends.md.txt

Illustrative examples of how to use the `torch.backends.python_native` module.

```APIDOC
## Usage Examples

```python
import torch.backends.python_native as pn

# Query available DSLs
print(pn.available_dsls)  # Example output: ['triton', 'cutedsl']

# Disable all Triton operations
pn.triton.enabled = False

# Temporarily disable CuteDSL operations
with pn.cutedsl.disabled():
    result = model(input)  # CuteDSL ops disabled

# Disable specific operations across all DSLs
pn.disable_operations('scaled_mm', '_flash_attention_forward')

# Query operations for a specific DSL
triton_ops = pn.get_dsl_operations('triton')
```
```

--------------------------------

### Example Input Metadata: pkg.torch.export.graph_signature.InputSpec.kind

Source: https://docs.pytorch.org/docs/2.12/_sources/onnx_export.md.txt

Lists example values for the 'pkg.torch.export.graph_signature.InputSpec.kind' metadata property.

```text
USER_INPUT
```

```text
PARAMETER
```

```text
BUFFER
```

```text
CONSTANT_TENSOR
```

```text
CUSTOM_OBJ
```

```text
TOKEN
```

--------------------------------

### Example Output Metadata: pkg.torch.export.graph_signature.OutputSpec.kind

Source: https://docs.pytorch.org/docs/2.12/_sources/onnx_export.md.txt

Lists example values for the 'pkg.torch.export.graph_signature.OutputSpec.kind' metadata property.

```text
USER_OUTPUT
```

```text
LOSS_OUTPUT
```

```text
BUFFER_MUTATION
```

```text
GRADIENT_TO_PARAMETER
```

```text
GRADIENT_TO_USER_INPUT
```

```text
USER_INPUT_MUTATION
```

```text
TOKEN
```

--------------------------------

### SequentialLR Example

Source: https://docs.pytorch.org/docs/2.12/generated/torch.optim.lr_scheduler.SequentialLR.html

Example demonstrating how to use SequentialLR with ConstantLR and ExponentialLR.

```APIDOC
```python
>>> # Assuming optimizer uses lr = 0.05 for all groups
>>> # lr = 0.005     if epoch == 0
>>> # lr = 0.005     if epoch == 1
>>> # lr = 0.005     if epoch == 2
>>> # ...
>>> # lr = 0.05      if epoch == 20
>>> # lr = 0.045     if epoch == 21
>>> # lr = 0.0405    if epoch == 22
>>> scheduler1 = ConstantLR(optimizer, factor=0.1, total_iters=20)
>>> scheduler2 = ExponentialLR(optimizer, gamma=0.9)
>>> scheduler = SequentialLR(
...     optimizer,
...     schedulers=[scheduler1, scheduler2],
...     milestones=[20],
... )
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
```
```

--------------------------------

### Set up Planner with Metadata

Source: https://docs.pytorch.org/docs/2.12/distributed.checkpoint.html

Example of setting up a planner by creating a Metadata object from a state dictionary.

```python
# Setups of the planner, extnding default behavior by creating the Metadata object from the state dict
```

--------------------------------

### Get the sign of tensor elements

Source: https://docs.pytorch.org/docs/2.12/generated/torch.sign.html

This example demonstrates how to use torch.sign to get the sign of each element in a tensor. The function returns a tensor of the same shape, where each element is 1, -1, or 0.

```python
>>> a = torch.tensor([0.7, -1.2, 0., 2.3])
>>> a
tensor([ 0.7000, -1.2000,  0.0000,  2.3000])
>>> torch.sign(a)
tensor([ 1., -1.,  0.,  1.])
```

--------------------------------

### Prepare Model for Quantization with prepare_fx

Source: https://docs.pytorch.org/docs/2.12/generated/torch.ao.quantization.quantize_fx.prepare_fx.html

This example demonstrates how to use `prepare_fx` to prepare a model for post-training quantization. It shows the initialization of a model, definition of a calibration function, configuration of quantization settings using `get_default_qconfig_mapping`, and the preparation of the model with example inputs.

```python
import torch
from torch.ao.quantization import get_default_qconfig_mapping
from torch.ao.quantization.quantize_fx import prepare_fx

class Submodule(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = torch.nn.Linear(5, 5)
    def forward(self, x):
        x = self.linear(x)
        return x

class M(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = torch.nn.Linear(5, 5)
        self.sub = Submodule()

    def forward(self, x):
        x = self.linear(x)
        x = self.sub(x) + x
        return x

# initialize a floating point model
float_model = M().eval()

# define calibration function
def calibrate(model, data_loader):
    model.eval()
    with torch.no_grad():
        for image, target in data_loader:
            model(image)

# qconfig is the configuration for how we insert observers for a particular
# operator
# qconfig = get_default_qconfig("fbgemm")
# Example of customizing qconfig:
# qconfig = torch.ao.quantization.QConfig(
#    activation=MinMaxObserver.with_args(dtype=torch.qint8),
#    weight=MinMaxObserver.with_args(dtype=torch.qint8))
# `activation` and `weight` are constructors of observer module

# qconfig_mapping is a collection of quantization configurations, user can
# set the qconfig for each operator (torch op calls, functional calls, module calls)
# in the model through qconfig_mapping
# the following call will get the qconfig_mapping that works best for models
# that target "fbgemm" backend
qconfig_mapping = get_default_qconfig_mapping("fbgemm")

# We can customize qconfig_mapping in different ways.
# e.g. set the global qconfig, which means we will use the same qconfig for
# all operators in the model, this can be overwritten by other settings
# qconfig_mapping = QConfigMapping().set_global(qconfig)
# e.g. quantize the linear submodule with a specific qconfig
# qconfig_mapping = QConfigMapping().set_module_name("linear", qconfig)
# e.g. quantize all nn.Linear modules with a specific qconfig
# qconfig_mapping = QConfigMapping().set_object_type(torch.nn.Linear, qconfig)
# for a more complete list, please see the docstring for :class:`torch.ao.quantization.QConfigMapping`
# argument

# example_inputs is a tuple of inputs, that is used to infer the type of the
# outputs in the model
# currently it's not used, but please make sure model(*example_inputs) runs
example_inputs = (torch.randn(1, 3, 224, 224),)

# TODO: add backend_config after we split the backend_config for fbgemm and qnnpack
# e.g. backend_config = get_default_backend_config("fbgemm")
# `prepare_fx` inserts observers in the model based on qconfig_mapping and
# backend_config. If the configuration for an operator in qconfig_mapping
# is supported in the backend_config (meaning it's supported by the target

```

--------------------------------

### Example of Guard Installation with Constant Input

Source: https://docs.pytorch.org/docs/2.12/_sources/user_guide/torch_compiler/torch.compiler_dynamo_deepdive.md.txt

This snippet demonstrates how a constant string input installs guards for type and value equality when using `@torch.compile`. Run with `TORCH_LOGS=guards` to see the generated guards.

```python
import torch

@torch.compile
def fn(a, b):
    return a * len(b)

fn(torch.arange(10), "Hello")
```

--------------------------------

### FractionalMaxPool2d Initialization Examples

Source: https://docs.pytorch.org/docs/2.12/generated/torch.nn.FractionalMaxPool2d.html

Demonstrates initializing FractionalMaxPool2d with different configurations for kernel size, output size, and output ratio. Exactly one of output_size or output_ratio must be defined.

```python
>>> # pool of square window of size=3, and target output size 13x12
>>> m = nn.FractionalMaxPool2d(3, output_size=(13, 12))
```

```python
>>> # pool of square window and target output size being half of input image size
>>> m = nn.FractionalMaxPool2d(3, output_ratio=(0.5, 0.5))
```

--------------------------------

### torch.distributed.elastic

Source: https://docs.pytorch.org/docs/2.12/torch.html

Comprehensive documentation for PyTorch Distributed Elastic, covering quickstart, training scripts, examples, and advanced features.

```APIDOC
## torch.distributed.elastic Quickstart

### Description
A quick start guide to using PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Train script

### Description
Information on training scripts compatible with PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Examples

### Description
Provides various examples demonstrating the usage of PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Elastic Agent

### Description
Details on the Elastic Agent component of PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Multiprocessing

### Description
Information regarding multiprocessing support within PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Error Propagation

### Description
Explains how errors are propagated in PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Rendezvous

### Description
Details on the rendezvous mechanism used by PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Expiration Timers

### Description
Information about expiration timers in PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Metrics

### Description
Documentation on metrics collection for PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Events

### Description
Details on events handled by PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Subprocess Handling

### Description
Information on how subprocesses are handled in PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Control Plane

### Description
Details on the control plane for PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## NUMA Binding

### Description
Information regarding NUMA binding in PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## Customization

### Description
Guidance on customizing PyTorch Distributed Elastic.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable

## TorchElastic Kubernetes

### Description
Information on deploying and running TorchElastic on Kubernetes.

### Method
Not specified (documentation reference)

### Endpoint
Not applicable
```

--------------------------------

### Symmetric Memory Pool Usage Example

Source: https://docs.pytorch.org/docs/2.12/symmetric_memory.html

An example demonstrating how to get a symmetric memory pool for a CUDA device, use it to allocate a tensor, and then perform a one-shot all-reduce operation on that tensor using a specified group name.

```python
pool = torch.distributed._symmetric_memory.get_mem_pool("cuda:0")
with torch.cuda.use_mem_pool(pool):
    tensor = torch.randn(1000, device="cuda:0")
tensor = torch.ops.symm_mem.one_shot_all_reduce(tensor, "sum", group_name)
```

--------------------------------

### Initialize and Use PrefixStore

Source: https://docs.pytorch.org/docs/2.12/distributed.html

Illustrates creating a PrefixStore by wrapping an existing store (e.g., TCPStore, FileStore, or HashStore) with a specified prefix. This prepends the prefix to all keys.

```python
import torch.distributed as dist
# Assuming 'store' is an initialized store object (e.g., TCPStore, FileStore, HashStore)
# Example with a placeholder store:
# store = dist.TCPStore("127.0.0.1", 1234, 2)
# prefix_store = dist.PrefixStore("my_prefix_", store)
# prefix_store.set("key", "value") # This will store 'my_prefix_key' in the underlying store
```

--------------------------------

### Direct Graph Manipulation Example

Source: https://docs.pytorch.org/docs/2.12/fx.html

Illustrates a basic example of direct graph manipulation by showing how to replace torch.add() calls with torch.mul() calls within a traced graph. This snippet serves as a starting point for more complex graph editing.

```python
import torch
import torch.fx

```

--------------------------------

### PrepareCustomConfig Example Usage

Source: https://docs.pytorch.org/docs/2.12/generated/torch.ao.quantization.fx.custom_config.PrepareCustomConfig.html

Demonstrates how to instantiate and configure PrepareCustomConfig by chaining various setter methods to customize quantization behavior for specific modules and graph elements.

```python
prepare_custom_config = PrepareCustomConfig()             .set_standalone_module_name("module1", qconfig_mapping, example_inputs,                 child_prepare_custom_config, backend_config)             .set_standalone_module_class(MyStandaloneModule, qconfig_mapping, example_inputs,                 child_prepare_custom_config, backend_config)             .set_float_to_observed_mapping(FloatCustomModule, ObservedCustomModule)             .set_non_traceable_module_names(["module2", "module3"])             .set_non_traceable_module_classes([NonTraceableModule1, NonTraceableModule2])             .set_input_quantized_indexes([0])             .set_output_quantized_indexes([0])             .set_preserved_attributes(["attr1", "attr2"])
```

--------------------------------

### CTCLoss with Un-padded Targets

Source: https://docs.pytorch.org/docs/2.12/generated/torch.nn.modules.loss.CTCLoss.html

Example demonstrating the usage of CTCLoss when targets are un-padded. This setup is for targets that are concatenated without padding.

```python
>>> # Target are to be un-padded
>>> T = 50  # Input sequence length
>>> C = 20  # Number of classes (including blank)
>>> N = 16  # Batch size
>>>
>>> # Initialize random batch of input vectors, for *size = (T,N,C)
>>> input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()
>>> input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
>>>
>>> # Initialize random batch of targets (0 = blank, 1:C = classes)
>>> target_lengths = torch.randint(low=1, high=T, size=(N,), dtype=torch.long)
```

--------------------------------

### main()

Source: https://docs.pytorch.org/docs/2.12/genindex.html

Entry point for distributed training or environment collection utilities.

```APIDOC
## main()

### Description
Entry point function, potentially used for launching distributed training jobs or collecting environment information.

### Module
- torch.distributed.launch
- torch.distributed.run
- torch.utils.collect_env
```

--------------------------------

### Start Processes with Binary Entrypoint

Source: https://docs.pytorch.org/docs/2.12/elastic/multiprocessing.html

Starts two copies of the '/usr/bin/touch' binary, each creating a different file. Provides empty environment dictionaries for both ranks. The log directory must exist and be empty.

```python
# ok; two copies of /usr/bin/touch: touch file1, touch file2
start_processes(
   name="trainer",
   entrypoint="/usr/bin/touch",
   args:{0:("file1",), 1:("file2",),
   envs:{0:{}, 1:{}},
   log_dir=log_dir
 )
```

--------------------------------

### Accessing the real part of a complex tensor

Source: https://docs.pytorch.org/docs/2.12/_sources/generated/torch.Tensor.real.rst.txt

This example demonstrates how to get the real part of a complex tensor using the `.real` attribute.

```APIDOC
## Accessing the real part of a complex tensor

### Description

This snippet shows how to retrieve the real component of a complex-valued tensor.

### Method

Attribute access

### Endpoint

`Tensor.real`

### Parameters

This attribute does not take any parameters.

### Request Example

```python
import torch

# Create a complex tensor
complex_tensor = torch.tensor([1+2j, 3+4j, 5+6j])

# Get the real part
real_part = complex_tensor.real

print(real_part)
```

### Response

#### Success Response

Returns a tensor containing the real parts of the input tensor.

#### Response Example

```
tensor([1., 3., 5.])
```
```

--------------------------------

### Flatten a Tensor with Specified Dimensions

Source: https://docs.pytorch.org/docs/2.12/generated/torch.flatten.html

This example shows how to flatten a tensor by specifying the start dimension. Only dimensions from start_dim to the end are flattened.

```python
>>> t = torch.tensor([[[1, 2],
...                    [3, 4]],
...                   [[5, 6],
...                    [7, 8]]])
>>> torch.flatten(t, start_dim=1)
tensor([[1, 2, 3, 4],
        [5, 6, 7, 8]])
```

--------------------------------

### init()

Source: https://docs.pytorch.org/docs/2.12/genindex.html

Initializes a specific backend or device.

```APIDOC
## init()

### Description
Initializes a specific backend or device.

### Method
(in module torch.cuda)
(in module torch.mtia)
(in module torch.xpu)
```

--------------------------------

### torch.slice_scatter Example 2

Source: https://docs.pytorch.org/docs/2.12/generated/torch.slice_scatter.html

Embeds an 8x2 tensor into an 8x8 tensor along dimension 1, with specific start, end, and step parameters.

```python
>>> b = torch.ones(8, 2)
>>> a.slice_scatter(b, dim=1, start=2, end=6, step=2)
tensor([[0., 0., 1., 0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 1., 0., 0., 0.]])
```

--------------------------------

### _set_up_planner

Source: https://docs.pytorch.org/docs/2.12/distributed.checkpoint.html

Abstract method called on every rank to initialize the planner with the state_dict, metadata, and coordinator status.

```APIDOC
_abstract _set_up_planner(_state_dict_ , _metadata =None_, _is_coordinator =False_)[source]#
    
Initialize this instance to load data into `state_dict`.
. N.B. This is called on every rank.
```

--------------------------------

### Higher-Precision Reduction Example Setup

Source: https://docs.pytorch.org/docs/2.12/_sources/symmetric_memory.md.txt

This snippet shows the necessary imports for utilizing higher-precision reduction with symmetric memory tensors in PyTorch distributed operations.

```python
import torch.distributed as dist
import torch.distributed._symmetric_memory as symm_mem
```

--------------------------------

### start_processes

Source: https://docs.pytorch.org/docs/2.12/elastic/multiprocessing.html

Starts `n` copies of `entrypoint` processes with the provided options. `entrypoint` can be a Callable (function) or a string (binary). The number of copies is determined by the number of entries for `args` and `envs` arguments, which need to have the same key set.

```APIDOC
## start_processes

### Description
Starts `n` copies of `entrypoint` processes with the provided options. `entrypoint` is either a `Callable` (function) or a `str` (binary). The number of copies is determined by the number of entries for `args` and `envs` arguments, which need to have the same key set. All local ranks must be accounted for. That is, the keyset should be `{0,1,...,(nprocs-1)}`.

### Method
`start_processes(_name_ , _entrypoint_ , _args_ , _envs_ , _logs_specs_ , _log_line_prefixes =None_, _start_method ='spawn'_, _numa_options =None_, _duplicate_stdout_filters =None_, _duplicate_stderr_filters =None_)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters Details
- **_name_**: Name for the processes.
- **_entrypoint_**: Either a `Callable` (function) or a `str` (binary).
- **_args_**: Arguments to pass to the entrypoint, mapped by replica index (local rank).
- **_envs_**: Environment variables to pass to the entrypoint, mapped by replica index (local rank).
- **_logs_specs_**: Configuration for log redirection and filtering. Includes `redirects` and `tee` bitmasks specifying which std stream(s) to redirect to a log file in the `log_dir`. Valid mask values are defined in `Std`. Can be a map with the key as the local rank to specify redirect behavior for specific ranks.
- **_log_line_prefixes_**: Optional prefixes for log lines.
- **_start_method_**: The multiprocessing start method (default: 'spawn').
- **_numa_options_**: NUMA related options.
- **_duplicate_stdout_filters_**: Optional filters to duplicate stdout lines to a file.
- **_duplicate_stderr_filters_**: Optional filters to duplicate stderr lines to a file.

### Request Example
```python
from torch.distributed.elastic.multiprocessing import Std, start_processes

def trainer(a, b, c):
    pass  # train

ctx = start_processes(
    name="trainer",
    entrypoint=trainer,
    args={0: (1, 2, 3), 1: (4, 5, 6)},
    envs={0: {"LOCAL_RANK": 0}, 1: {"LOCAL_RANK": 1}},
    log_dir="/tmp/foobar",
    redirects=Std.ALL,  # write all worker stdout/stderr to a log file
    tee={0: Std.ERR},  # tee only local rank 0's stderr to console
)
cox.wait()
```

### Response
#### Success Response (200)
Returns a process context (`api.PContext`). If a function was launched, `api.MultiprocessContext` is returned. If a binary was launched, `api.SubprocessContext` is returned. Both are specific implementations of the parent `api.PContext` class.

#### Response Example
None provided in source.
```

--------------------------------

### Transitive Dictionary Get Example

Source: https://docs.pytorch.org/docs/2.12/generated/torch.fx.experimental.unification.utils.transitive_get.html

Demonstrates how transitive_get retrieves a value from a nested dictionary structure. This is useful for accessing values through multiple levels of dictionary keys.

```python
d = {1: 2, 2: 3, 3: 4}
print(d.get(1))
print(transitive_get(1, d))
```

--------------------------------

### Create a tensor using keyword arguments

Source: https://docs.pytorch.org/docs/2.12/generated/torch.linspace.html

Achieves the same result as the previous example by explicitly naming the 'start', 'end', and 'steps' arguments. This can improve code readability.

```python
>>> torch.linspace(start=-10, end=10, steps=5)
tensor([-10.,  -5.,   0.,   5.,  10.])
```

--------------------------------

### Example QConfigMapping Usage

Source: https://docs.pytorch.org/docs/2.12/generated/torch.ao.quantization.qconfig_mapping.QConfigMapping.html

Demonstrates how to initialize a QConfigMapping and set various QConfigs using different methods like set_global, set_object_type, set_module_name_regex, set_module_name, and set_module_name_object_type_order.

```python
qconfig_mapping = QConfigMapping()
    .set_global(global_qconfig)
    .set_object_type(torch.nn.Linear, qconfig1)
    .set_object_type(torch.nn.ReLU, qconfig1)
    .set_module_name_regex("foo.*bar.*conv[0-9]+", qconfig1)
    .set_module_name_regex("foo.*", qconfig2)
    .set_module_name("module1", qconfig1)
    .set_module_name("module2", qconfig2)
    .set_module_name_object_type_order("foo.bar", torch.nn.functional.linear, 0, qconfig3)

```

--------------------------------

### DDP Initialization Logs

Source: https://docs.pytorch.org/docs/2.12/_sources/distributed.md.txt

Example of debug logs rendered at initialization time when TORCH_DISTRIBUTED_DEBUG is set to INFO or DETAIL. These logs provide details about the DDP setup.

```log
I0607 16:10:35.739390 515217 logger.cpp:173] [Rank 0]: DDP Initialized with:
broadcast_buffers: 1
bucket_cap_bytes: 26214400
find_unused_parameters: 0
gradient_as_bucket_view: 0
is_multi_device_module: 0
iteration: 0
num_parameter_tensors: 2
output_device: 0
rank: 0
total_parameter_size_bytes: 440
backend_name: nccl
bucket_sizes: 440
cuda_visible_devices: N/A
device_ids: 0
dtypes: float
master_addr: localhost
master_port: 29501
module_name: TwoLinLayerNet
nccl_async_error_handling: N/A
nccl_blocking_wait: N/A
nccl_debug: WARN
nccl_ib_timeout: N/A
nccl_nthreads: N/A
nccl_socket_ifname: N/A
torch_distributed_debug: INFO

```

--------------------------------

### _set_up_storage_reader (Load)

Source: https://docs.pytorch.org/docs/2.12/distributed.checkpoint.html

Initializes the storage reader with metadata and coordinator status.

```APIDOC
## _set_up_storage_reader (Load)

### Description
Initializes this instance with the provided metadata and coordinator status.

### Parameters
* **metadata** (Metadata) – The metadata schema to use.
* **is_coordinator** (bool) – Indicates whether this instance is responsible for coordinating the checkpoint.
```

--------------------------------

### Example of Dynamo Guards with Constant String Input

Source: https://docs.pytorch.org/docs/2.12/user_guide/torch_compiler/torch.compiler_dynamo_deepdive.html

Demonstrates how Dynamo installs guards for constant inputs like strings. Running with TORCH_LOGS=guards shows the generated checks.

```python
import torch

@torch.compile
def fn(a, b):
    return a * len(b)

fn(torch.arange(10), "Hello")

```

--------------------------------

### Abstract _set_up_storage_reader Method

Source: https://docs.pytorch.org/docs/2.12/distributed.checkpoint.html

Initializes the storage reader instance with metadata and coordinator status.

```python
def _set_up_storage_reader(self, metadata, is_coordinator, *args, **kwargs):
    """
    Initialize this instance. 

    Parameters:
        metadata (Metadata): The metadata schema to use.
        is_coordinator (bool): Whether this instance is responsible for coordinating the checkpoint.
    """
    pass
```

--------------------------------

### Plotting with torch.functional.meshgrid and 'xy' indexing

Source: https://docs.pytorch.org/docs/2.12/generated/torch.functional.meshgrid.html

Generates a grid for plotting using 'xy' indexing, commonly used with libraries like Matplotlib. Ensure Matplotlib is installed for this example.

```python
>>> import matplotlib.pyplot as plt
>>> xs = torch.linspace(-5, 5, steps=100)
>>> ys = torch.linspace(-5, 5, steps=100)
>>> x, y = torch.meshgrid(xs, ys, indexing='xy')
>>> z = torch.sin(torch.sqrt(x * x + y * y))
>>> ax = plt.axes(projection='3d')
>>> ax.plot_surface(x.numpy(), y.numpy(), z.numpy())
>>> plt.show()
```

--------------------------------

### ConvTranspose3d Initialization Examples

Source: https://docs.pytorch.org/docs/2.12/generated/torch.nn.ConvTranspose3d.html

Demonstrates initializing ConvTranspose3d with different kernel sizes, strides, and padding. The first example uses square kernels and equal strides, while the second uses non-square kernels, unequal strides, and padding.

```python
>>> # With square kernels and equal stride
>>> m = nn.ConvTranspose3d(16, 33, 3, stride=2)
```

```python
>>> # non-square kernels and unequal stride and with padding
>>> m = nn.ConvTranspose3d(16, 33, (3, 5, 2), stride=(2, 1, 1), padding=(0, 4, 2))
```

```python
>>> input = torch.randn(20, 16, 10, 50, 100)
>>> output = m(input)
```

--------------------------------

### AvgPool3d Initialization Examples

Source: https://docs.pytorch.org/docs/2.12/generated/torch.nn.AvgPool3d.html

Demonstrates initializing AvgPool3d with a square kernel size and stride, and with a non-square kernel size and stride. An example input tensor is also shown.

```python
>>> # pool of square window of size=3, stride=2
>>> m = nn.AvgPool3d(3, stride=2)
>>> # pool of non-square window
>>> m = nn.AvgPool3d((3, 2, 2), stride=(2, 1, 2))
>>> input = torch.randn(20, 16, 50, 44, 31)
>>> output = m(input)
```

--------------------------------

### Sequential Learning Rate Scheduler Example

Source: https://docs.pytorch.org/docs/2.12/generated/torch.optim.Adamax.html

Demonstrates setting up and loading state for a sequential learning rate scheduler composed of LinearLR and CosineAnnealingLR. This example shows how to load both the scheduler and optimizer state dictionaries.

```python
>>> model = torch.nn.Linear(10, 10)
>>> optim = torch.optim.SGD(model.parameters(), lr=3e-4)
>>> scheduler1 = torch.optim.lr_scheduler.LinearLR(
...     optim,
...     start_factor=0.1,
...     end_factor=1,
...     total_iters=20,
... )
>>> scheduler2 = torch.optim.lr_scheduler.CosineAnnealingLR(
...     optim,
...     T_max=80,
...     eta_min=3e-5,
... )
>>> lr = torch.optim.lr_scheduler.SequentialLR(
...     optim,
...     schedulers=[scheduler1, scheduler2],
...     milestones=[20],
... )
>>> lr.load_state_dict(torch.load("./save_seq.pt"))
>>> # now load the optimizer checkpoint after loading the LRScheduler
>>> optim.load_state_dict(torch.load("./save_optim.pt"))

```

--------------------------------

### Get Tensor Storage Offset

Source: https://docs.pytorch.org/docs/2.12/generated/torch.Tensor.storage_offset.html

Demonstrates how to retrieve the storage offset for a tensor and its slice. The offset indicates the starting position of the tensor's data within the shared storage.

```python
>>> x = torch.tensor([1, 2, 3, 4, 5])
>>> x.storage_offset()
0
>>> x[3:].storage_offset()
3
```

--------------------------------

### Initialize and Use FileStore

Source: https://docs.pytorch.org/docs/2.12/distributed.html

Demonstrates initializing FileStore on two processes using the same file path. This allows for inter-process communication via a file on disk.

```python
import torch.distributed as dist
store1 = dist.FileStore("/tmp/filestore", 2)
store2 = dist.FileStore("/tmp/filestore", 2)
# Use any of the store methods from either the client or server after initialization
store1.set("first_key", "first_value")
store2.get("first_key")
```

--------------------------------

### Basic DDP Example

Source: https://docs.pytorch.org/docs/2.12/notes/ddp.html

This example demonstrates a basic setup for Distributed Data Parallel training using PyTorch. It initializes a process group, defines a local model, wraps it with DDP, and performs a forward pass, backward pass, and optimizer step. Ensure environment variables MASTER_ADDR and MASTER_PORT are set for distributed initialization.

```python
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
import os
from torch.nn.parallel import DistributedDataParallel as DDP


def example(rank, world_size):
    # create default process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)
    # create local model
    model = nn.Linear(10, 10).to(rank)
    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # forward pass
    outputs = ddp_model(torch.randn(20, 10).to(rank))
    labels = torch.randn(20, 10).to(rank)
    # backward pass
    loss_fn(outputs, labels).backward()
    # update parameters
    optimizer.step()

def main():
    world_size = 2
    mp.spawn(example,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main":
    # Environment variables which need to be
    # set when using c10d's default "env"
    # initialization mode.
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "29500"
    main()

```

--------------------------------

### Conv3d Initialization Examples

Source: https://docs.pytorch.org/docs/2.12/generated/torch.nn.Conv3d.html

Demonstrates initializing Conv3d with square kernels and equal stride, and with non-square kernels, unequal stride, and padding. An example input tensor is also shown.

```python
>>> # With square kernels and equal stride
>>> m = nn.Conv3d(16, 33, 3, stride=2)
>>> # non-square kernels and unequal stride and with padding
>>> m = nn.Conv3d(16, 33, (3, 5, 2), stride=(2, 1, 1), padding=(4, 2, 0))
>>> input = torch.randn(20, 16, 10, 50, 100)
>>> output = m(input)
```

--------------------------------

### SequentialLR Example

Source: https://docs.pytorch.org/docs/2.12/generated/torch.optim.lr_scheduler.SequentialLR.html

Demonstrates how to use SequentialLR to apply a ConstantLR for the first 20 epochs and then an ExponentialLR for subsequent epochs. The learning rate starts at 0.005 and decays exponentially after epoch 20.

```python
>>> # Assuming optimizer uses lr = 0.05 for all groups
>>> # lr = 0.005     if epoch == 0
>>> # lr = 0.005     if epoch == 1
>>> # lr = 0.005     if epoch == 2
>>> # ...
>>> # lr = 0.05      if epoch == 20
>>> # lr = 0.045     if epoch == 21
>>> # lr = 0.0405    if epoch == 22
>>> scheduler1 = ConstantLR(optimizer, factor=0.1, total_iters=20)
>>> scheduler2 = ExponentialLR(optimizer, gamma=0.9)
>>> scheduler = SequentialLR(
...     optimizer,
...     schedulers=[scheduler1, scheduler2],
...     milestones=[20],
... )
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
```

--------------------------------

### RPC Sync Example for Distributed Model

Source: https://docs.pytorch.org/docs/2.12/_sources/rpc/distributed_autograd.rst.txt

Demonstrates a simple distributed model setup using RPC, where a function is called synchronously on a remote worker. Ensure RPC is initialized before use.

```python
import torch
import torch.distributed.rpc as rpc

def my_add(t1, t2):
  return torch.add(t1, t2)

# On worker 0:
t1 = torch.rand((3, 3), requires_grad=True)
t2 = torch.rand((3, 3), requires_grad=True)

# Perform some computation remotely.
t3 = rpc.rpc_sync("worker1", my_add, args=(t1, t2))

# Perform some computation locally based on remote result.
t4 = torch.rand((3, 3), requires_grad=True)
t5 = torch.mul(t3, t4)

# Compute some loss.
loss = t5.sum()
```

--------------------------------

### Start Processes with Missing Environment Variables

Source: https://docs.pytorch.org/docs/2.12/elastic/multiprocessing.html

Example demonstrating an invalid configuration where environment variables are missing for local rank 1. All local ranks must have corresponding entries in the 'envs' dictionary.

```python
# invalid; envs missing for local rank 1
start_processes(
   name="trainer",
   entrypoint=foo,
   args:{0:("bar0",), 1:("bar1",),
   envs:{0:{}},
   log_dir=log_dir
)
```

--------------------------------

### SequentialLR Example with Optimizer and Scheduler State Loading

Source: https://docs.pytorch.org/docs/2.12/generated/torch.optim.Adadelta.html

Demonstrates loading state dictionaries for a SequentialLR scheduler, an optimizer, and a CosineAnnealingLR scheduler. This example highlights the order of loading state dictionaries when using multiple schedulers and an optimizer.

```python
>>> model = torch.nn.Linear(10, 10)
>>> optim = torch.optim.SGD(model.parameters(), lr=3e-4)
>>> scheduler1 = torch.optim.lr_scheduler.LinearLR(
...     optim,
...     start_factor=0.1,
...     end_factor=1,
...     total_iters=20,
... )
>>> scheduler2 = torch.optim.lr_scheduler.CosineAnnealingLR(
...     optim,
...     T_max=80,
...     eta_min=3e-5,
... )
>>> lr = torch.optim.lr_scheduler.SequentialLR(
...     optim,
...     schedulers=[scheduler1, scheduler2],
...     milestones=[20],
... )
>>> lr.load_state_dict(torch.load("./save_seq.pt"))
>>> # now load the optimizer checkpoint after loading the LRScheduler
>>> optim.load_state_dict(torch.load("./save_optim.pt"))


```

--------------------------------

### torch.argsort Example

Source: https://docs.pytorch.org/docs/2.12/generated/torch.argsort.html

Demonstrates how to use torch.argsort to get the indices that sort a tensor along a specified dimension. The `stable` argument can be set to `True` for a stable sort, which preserves the order of equivalent elements but is slower.

```python
>>> a = torch.randn(4, 4)
>>> a
tensor([[ 0.0785,  1.5267, -0.8521,  0.4065],
        [ 0.1598,  0.0788, -0.0745, -1.2700],
        [ 1.2208,  1.0722, -0.7064,  1.2564],
        [ 0.0669, -0.2318, -0.8229, -0.9280]])


>>> torch.argsort(a, dim=1)
tensor([[2, 0, 3, 1],
        [3, 2, 1, 0],
        [2, 1, 0, 3],
        [3, 2, 1, 0]])
```

--------------------------------

### Create C++ Extension with setuptools

Source: https://docs.pytorch.org/docs/2.12/cpp_extension.html

Use CppExtension to create a setuptools.Extension for C++ code. All arguments are forwarded to the setuptools.Extension constructor.

```python
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CppExtension
setup(
    name='extension',
    ext_modules=[
        CppExtension(
            name='extension',
            sources=['extension.cpp'],
            extra_compile_args=['-g'],
            extra_link_args=['-Wl,--no-as-needed', '-lm'])
    ],
    cmdclass={
        'build_ext': BuildExtension
    })

```

--------------------------------

### Training with FP32 on Intel GPU

Source: https://docs.pytorch.org/docs/2.12/_sources/notes/get_start_xpu.rst.txt

Example of a training workflow using the CIFAR10 dataset and a ResNet50 model on an Intel GPU with FP32 precision. It includes data loading, model setup, loss calculation, and optimization steps.

```python
import torch
import torchvision

LR = 0.001
DOWNLOAD = True
DATA = "datasets/cifar10/"

transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize((224, 224)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    ]
)
train_dataset = torchvision.datasets.CIFAR10(
    root=DATA,
    train=True,
    transform=transform,
    download=DOWNLOAD,
)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=128)
train_len = len(train_loader)

model = torchvision.models.resnet50()
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=LR, momentum=0.9)
model.train()
model = model.to("xpu")
criterion = criterion.to("xpu")

print(f"Initiating training")
for batch_idx, (data, target) in enumerate(train_loader):
    data = data.to("xpu")
    target = target.to("xpu")
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    if (batch_idx + 1) % 10 == 0:
        iteration_loss = loss.item()
        print(f"Iteration [{batch_idx+1}/{train_len}], Loss: {iteration_loss:.4f}")
torch.save(
    {

```

--------------------------------

### Start Processes with Function Entrypoint

Source: https://docs.pytorch.org/docs/2.12/elastic/multiprocessing.html

Starts two copies of a function 'foo' with different string arguments. Ensures environment variables are provided for all local ranks. The log directory must exist and be empty.

```python
log_dir = "/tmp/test"

# ok; two copies of foo: foo("bar0"), foo("bar1")
start_processes(
   name="trainer",
   entrypoint=foo,
   args:{0:("bar0",), 1:("bar1",),
   envs:{0:{}, 1:{}},
   log_dir=log_dir
)
```

--------------------------------

### collect_producer_nodes

Source: https://docs.pytorch.org/docs/2.12/generated/torch.ao.quantization.fx.utils.collect_producer_nodes.html

Starting from a target node, trace back until we hit an input or getattr node. This is used to extract the chain of operators starting from getattr to the target node. For example, in a `forward` method like `observed = self.observer(self.weight); return F.linear(x, observed)`, `collect_producer_nodes(observed)` will return a list of nodes that produce the `observed` node, or `None` if a self-contained graph cannot be extracted without free variables.

```APIDOC
class _torch.ao.quantization.fx.utils.collect_producer_nodes(_node_)

    Starting from a target node, trace back until we hit input or getattr node. This is used to extract the chain of operators starting from getattr to the target node, for example:
    ```
    def forward(self, x):
        observed = self.observer(self.weight)
        return F.linear(x, observed)

    ```

    collect_producer_nodes(observed) will either return a list of nodes that produces the observed node or None if we can’t extract a self contained graph without free variables(inputs of the forward function).

    Return type:
        list[_Node_] | None
```

--------------------------------

### main()

Source: https://docs.pytorch.org/docs/2.12/elastic/generated/torch.distributed.run.main.html

The main entry point for launching distributed training jobs. It accepts an optional list of arguments.

```APIDOC
## main(_args =None_)

### Description
This function serves as the primary entry point for executing distributed training scripts. It can optionally accept a list of arguments to configure its behavior.

### Signature
`torch.distributed.run.main(_args =None_)`

### Parameters
* `_args` (list, optional): A list of strings representing command-line arguments. Defaults to None, in which case arguments are read from `sys.argv`.
```

--------------------------------

### Basic DDP Example with Linear Model

Source: https://docs.pytorch.org/docs/2.12/_sources/notes/ddp.rst.txt

Demonstrates a fundamental Distributed Data Parallel setup using a linear model, including initialization, forward/backward passes, and optimizer steps. Ensure environment variables for master address and port are set.

```python
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
import os
from torch.nn.parallel import DistributedDataParallel as DDP


def example(rank, world_size):
    # create default process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)
    # create local model
    model = nn.Linear(10, 10).to(rank)
    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # forward pass
    outputs = ddp_model(torch.randn(20, 10).to(rank))
    labels = torch.randn(20, 10).to(rank)
    # backward pass
    loss_fn(outputs, labels).backward()
    # update parameters
    optimizer.step()

def main():
    world_size = 2
    mp.spawn(example,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    # Environment variables which need to be
    # set when using c10d's default "env"
    # initialization mode.
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "29500"
    main()
```

--------------------------------

### Async Function with Chained RPC

Source: https://docs.pytorch.org/docs/2.12/rpc.html

Use this decorator when a function's return value is a `Future` and it needs to execute asynchronously on the RPC callee. The callee installs subsequent processing as a callback to the `Future`. This example shows chaining an `rpc_async` call with a `then` callback.

```python
from torch.distributed import rpc

# omitting setup and shutdown RPC

# On all workers
@rpc.functions.async_execution
def async_add_chained(to, x, y, z):
    # This function runs on "worker1" and returns immediately when
    # the callback is installed through the `then(cb)` API. In the
    # mean time, the `rpc_async` to "worker2" can run concurrently.
    # When the return value of that `rpc_async` arrives at
    # "worker1", "worker1" will run the lambda function accordingly
    # and set the value for the previously returned `Future`, which
    # will then trigger RPC to send the result back to "worker0".
    return rpc.rpc_async(to, torch.add, args=(x, y)).then(
        lambda fut: fut.wait() + z
    )

# On worker0
ret = rpc.rpc_sync(
    "worker1",
    async_add_chained,
    args=("worker2", torch.ones(2), 1, 1)
)
print(ret)  # prints tensor([3., 3.])

```