### Install and run TensorBoard

Source: https://docs.pytorch.org/docs/stable/_sources/tensorboard.rst.txt

Commands to install the TensorBoard package and launch the server to view logged data.

```bash
pip install tensorboard
tensorboard --logdir=runs
```

--------------------------------

### Running Training Example via Command Line

Source: https://docs.pytorch.org/docs/stable/notes/multiprocessing.html

Command to execute the training example on CPU using a specified number of processes. Be aware that running this with multiple processes can lead to CPU oversubscription if not managed carefully.

```bash
python main.py --num-processes 4

```

--------------------------------

### Conv2d Initialization Examples

Source: https://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html

Demonstrates initializing Conv2d with different parameters. Use these examples to set up convolutional layers with specific kernel shapes, strides, padding, and dilation.

```python
m = nn.Conv2d(16, 33, 3, stride=2)
```

```python
m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
```

```python
m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2), dilation=(3, 1))
```

--------------------------------

### FSDP Initialization Example

Source: https://docs.pytorch.org/docs/stable/fsdp.html

Example demonstrating how to initialize an FSDP model with custom parameter initialization functions and deferred initialization.

```APIDOC
## FSDP Initialization Example

### Request Example
```python
>>> module = MyModule(device="meta")
>>> def my_init_fn(module: nn.Module):
>>>     # E.g. initialize depending on the module type
>>>     ...
>>> fsdp_model = FSDP(module, param_init_fn=my_init_fn, auto_wrap_policy=size_based_auto_wrap_policy)
>>> print(next(fsdp_model.parameters()).device) # current CUDA device
>>> # With torchdistX
>>> module = deferred_init.deferred_init(MyModule, device="cuda")
>>> # Will initialize via deferred_init.materialize_module().
>>> fsdp_model = FSDP(module, auto_wrap_policy=size_based_auto_wrap_policy)
```
```

--------------------------------

### CTCLoss Implementation Examples

Source: https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.loss.CTCLoss.html

Examples demonstrating the initialization and usage of CTCLoss with both batched and unbatched target sequences.

```python
>>> target = torch.randint(
...     low=1,
...     high=C,
...     size=(sum(target_lengths),),
...     dtype=torch.long,
... )
>>> ctc_loss = nn.CTCLoss()
>>> loss = ctc_loss(input, target, input_lengths, target_lengths)
>>> loss.backward()
```

```python
>>> # Target are to be un-padded and unbatched (effectively N=1)
>>> T = 50  # Input sequence length
>>> C = 20  # Number of classes (including blank)
>>>
>>> # Initialize random batch of input vectors, for *size = (T,C)
>>> input = torch.randn(T, C).log_softmax(1).detach().requires_grad_()
>>> input_lengths = torch.tensor(T, dtype=torch.long)
>>>
>>> # Initialize random batch of targets (0 = blank, 1:C = classes)
>>> target_lengths = torch.randint(low=1, high=T, size=(), dtype=torch.long)
>>> target = torch.randint(
...     low=1,
...     high=C,
...     size=(target_lengths,),
...     dtype=torch.long,
... )
>>> ctc_loss = nn.CTCLoss()
>>> loss = ctc_loss(input, target, input_lengths, target_lengths)
>>> loss.backward()
```

--------------------------------

### Example Usage of SparseAdam

Source: https://docs.pytorch.org/docs/stable/generated/torch.optim.SparseAdam.html

Illustrative example demonstrating the initialization of a model and optimizer. Note: This is a partial example from the documentation.

```python
>>> model = torch.nn.Linear(10, 10)
```

--------------------------------

### Install Nightly PyTorch for Intel GPU

Source: https://docs.pytorch.org/docs/stable/_sources/notes/get_start_xpu.rst.txt

Installs the latest preview/nightly wheels for Intel GPU (XPU).

```bash
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/xpu
```

--------------------------------

### torch.hub.help

Source: https://docs.pytorch.org/docs/stable/_sources/hub.md.txt

Shows the docstring and examples for a specific model entrypoint.

```APIDOC
## GET torch.hub.help

### Description
Displays the docstring and usage examples for a specific model entrypoint within a repository.

### Method
GET

### Endpoint
torch.hub.help(repo_or_dir, model, source='github', force_reload=False)
```

--------------------------------

### GRU Usage Example

Source: https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.rnn.GRU.html

A basic example demonstrating how to use the GRU module.

```APIDOC
## GRU Usage Example

### Request Example
```python
import torch
import torch.nn as nn

rnn = nn.GRU(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
output, hn = rnn(input, h0)
```

### Response Example
- **output** (Tensor): The output features (h_t) from the last layer of the GRU, for each t. Shape: (seq_len, batch, num_directions * hidden_size) if batch_first=False, otherwise (batch, seq_len, num_directions * hidden_size).
- **hn** (Tensor): The hidden state for t=seq_len. Shape: (num_layers * num_directions, batch, hidden_size).
```

--------------------------------

### Combined Forward and Context Setup

Source: https://docs.pytorch.org/docs/stable/generated/torch.autograd.Function.forward.html

Use this method when the forward pass and context setup can be combined into a single static method. It accepts a context object `ctx` as the first argument.

```python
@staticmethod
def forward(ctx: Any, *args: Any, **kwargs: Any) -> Any:
    pass
```

--------------------------------

### RRef Examples

Source: https://docs.pytorch.org/docs/stable/rpc.html

Examples demonstrating how to create and use RRefs, including creating from remote calls and local objects, and sharing them across workers.

```APIDOC
### Example: Create an RRef using rpc.remote

```python
import torch
import torch.distributed.rpc as rpc
rref = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3))
# get a copy of value from the RRef
x = rref.to_here()
```

### Example: Create an RRef from a local object

```python
import torch
from torch.distributed.rpc import RRef
x = torch.zeros(2, 2)
rref = RRef(x)
```

### Example: Share an RRef with other workers

```python
# On both worker0 and worker1:
def f(rref):
  return rref.to_here() + 1

# On worker0:
import torch
import torch.distributed.rpc as rpc
from torch.distributed.rpc import RRef
rref = RRef(torch.zeros(2, 2))
# the following RPC shares the rref with worker1, reference
# count is automatically updated.
rpc.rpc_sync("worker1", f, args=(rref,))
```
```

--------------------------------

### Define tensors for comparison

Source: https://docs.pytorch.org/docs/stable/testing.html

Example setup for tensor comparison.

```python
>>> expected = torch.tensor([1.0, 2.0, 3.0])
>>> actual = torch.tensor([1.0, 4.0, 5.0])
```

--------------------------------

### Testing Tutorial Examples

Source: https://docs.pytorch.org/docs/stable/_sources/notes/local_tensor_tutorial.rst.txt

Information on the test suite used to verify the correctness of the tutorial examples.

```APIDOC
## Testing Tutorial Examples

### Description
This section outlines the test suite used to ensure the accuracy and functionality of the provided PyTorch distributed tensor tutorial examples. The tests directly invoke the example functions.

### Test Suite Example
```python
# From test_local_tensor_tutorial_examples.py
from example_01_basic_operations import create_local_tensor

def test_create_local_tensor(self):
    lt = create_local_tensor()
    self.assertIsInstance(lt, LocalTensor)
    self.assertEqual(lt.shape, torch.Size([2, 2]))
```

### Test Suite Source
`test_local_tensor_tutorial_examples.py <https://github.com/pytorch/pytorch/blob/main/test/distributed/local_tensor_tutorial_examples/test_local_tensor_tutorial_examples.py>`_
```

--------------------------------

### example_arguments()

Source: https://docs.pytorch.org/docs/stable/onnx_export.html

Provides sample inputs for tracing, testing, and ONNX export.

```APIDOC
## example_arguments()

### Description
Return example arguments for the model's forward method. This method must be implemented by subclasses to provide sample inputs.

### Returns
- **Arguments** (tuple) - A tuple containing positional arguments and an optional dictionary of keyword arguments.
```

--------------------------------

### Get Symmetric Memory Pool with Usage Example

Source: https://docs.pytorch.org/docs/stable/symmetric_memory.html

Retrieves or creates a symmetric memory pool for a given device. Tensors allocated with this pool must be symmetric across ranks and can be used with symmetric operations. The example demonstrates getting a pool, using it for tensor allocation, and then performing an all-reduce operation.

```python
>>> pool = torch.distributed._symmetric_memory.get_mem_pool("cuda:0")
>>> with torch.cuda.use_mem_pool(pool):
>>>     tensor = torch.randn(1000, device="cuda:0")
>>> tensor = torch.ops.symm_mem.one_shot_all_reduce(tensor, "sum", group_name)
```

--------------------------------

### Reduce-Scatter Tensor Initialization

Source: https://docs.pytorch.org/docs/stable/distributed.html

Example setup for reduce-scatter operations using CUDA tensors.

```python
>>> # All tensors below are of torch.int64 dtype and on CUDA devices.
```

--------------------------------

### Initialize PrefixStore

Source: https://docs.pytorch.org/docs/stable/distributed.html

Demonstrates creating a PrefixStore which wraps another store and prepends a prefix to all keys.

```python
import torch.distributed as dist
# Assuming 'store' is an initialized store object like TCPStore, FileStore, or HashStore
# prefix_store = dist.PrefixStore("my_prefix_", store)
```

--------------------------------

### Example Callgrind Output with Path Prefixes

Source: https://docs.pytorch.org/docs/stable/benchmark_utils.html

This example illustrates how path prefixes in Callgrind output can complicate diffing. Stripping these prefixes can improve the accuracy of comparisons.

```text
23234231 /tmp/first_build_dir/thing.c:foo(...)
 9823794 /tmp/first_build_dir/thing.c:bar(...)
  ...
   53453 .../aten/src/Aten/...:function_that_actually_changed(...)
  ...
 -9823794 /tmp/second_build_dir/thing.c:bar(...)
-23234231 /tmp/second_build_dir/thing.c:foo(...)

```

--------------------------------

### GRU Usage Example

Source: https://docs.pytorch.org/docs/stable/generated/torch.ao.nn.quantized.dynamic.GRU.html

A basic example demonstrating how to initialize and use the GRU module with random input and hidden states.

```APIDOC
## GRU Usage Example

### Description
This example shows how to create a GRU layer and pass input data and an initial hidden state through it.

### Request Example
```python
import torch
import torch.nn as nn

rnn = nn.GRU(10, 20, 2)
input = torch.randn(5, 3, 10) # Input tensor of shape (seq_len, batch, input_size)
h0 = torch.randn(2, 3, 20) # Initial hidden state of shape (num_layers, batch, hidden_size)

output, hn = rnn(input, h0)

print("Output shape:", output.shape)
print("Hidden state shape:", hn.shape)
```

### Response
#### Success Response (200)
- **output** (Tensor) - The output features (h_t) from the last layer of the GRU, of shape (seq_len, batch, hidden_size).
- **hn** (Tensor) - The final hidden state for each element in the batch, of shape (num_layers, batch, hidden_size).

#### Response Example
```
Output shape: torch.Size([5, 3, 20])
Hidden state shape: torch.Size([2, 3, 20])
```
```

--------------------------------

### Initialize and Use TCPStore

Source: https://docs.pytorch.org/docs/stable/distributed.html

Demonstrates initializing a TCPStore as a server and a client, then setting and getting a value.

```python
server_store = dist.TCPStore("127.0.0.1", 1234, 2, True, timedelta(seconds=30))
# Run on process 2 (client)
client_store = dist.TCPStore("127.0.0.1", 1234, 2, False)
# Use any of the store methods from either the client or server after initialization
server_store.set("first_key", "first_value")
client_store.get("first_key")
```

--------------------------------

### Example of Dynamo Guards with Constant String Input

Source: https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/torch.compiler_dynamo_deepdive.html

This example demonstrates how Dynamo installs guards for constant inputs like strings. Running with TORCH_LOGS=guards shows the generated guards for type and value equality.

```python
import torch

@torch.compile
def fn(a, b):
    return a * len(b)

fn(torch.arange(10), "Hello")

```

--------------------------------

### Configuring Module Initialization

Source: https://docs.pytorch.org/docs/stable/_sources/notes/modules.rst.txt

Examples of initializing modules with specific devices, dtypes, or custom initialization logic.

```python
# Initialize module directly onto GPU.
   m = nn.Linear(5, 3, device='cuda')

   # Initialize module with 16-bit floating point parameters.
   m = nn.Linear(5, 3, dtype=torch.half)

   # Skip default parameter initialization and perform custom (e.g. orthogonal) initialization.
   m = torch.nn.utils.skip_init(nn.Linear, 5, 3)
   nn.init.orthogonal_(m.weight)
```

--------------------------------

### Hogwild Asynchronous Training Example

Source: https://docs.pytorch.org/docs/stable/notes/multiprocessing.html

Minimal example demonstrating the structure for Hogwild asynchronous training. Ensure global statements are guarded with `if __name__ == '__main__'` to prevent execution in all subprocesses, especially when using start methods other than 'fork'.

```python
import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        optimizer.step()  # This will update the shared parameters

if __name__ == '__main__':
    num_processes = 4
    model = MyModel()
    # NOTE: this is required for the ``fork`` method to work
    model.share_memory()
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=train, args=(model,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()

```

--------------------------------

### Generate evenly spaced tensors with torch.linspace

Source: https://docs.pytorch.org/docs/stable/generated/torch.linspace.html

Examples demonstrating how to create tensors with different start, end, and step configurations.

```python
>>> torch.linspace(3, 10, steps=5)
tensor([  3.0000,   4.7500,   6.5000,   8.2500,  10.0000])
>>> torch.linspace(-10, 10, steps=5)
tensor([-10.,  -5.,   0.,   5.,  10.])
>>> torch.linspace(start=-10, end=10, steps=5)
tensor([-10.,  -5.,   0.,   5.,  10.])
>>> torch.linspace(start=-10, end=10, steps=1)
tensor([-10.])
```

--------------------------------

### Get the first element of a sequence

Source: https://docs.pytorch.org/docs/stable/generated/torch.fx.experimental.unification.unification_tools.first.html

Use this function to access the initial item in any iterable sequence. No special setup is required.

```python
>>> first("ABC")
'A'
```

--------------------------------

### Launch Multi-Node Multi-Process Training (Node 1)

Source: https://docs.pytorch.org/docs/stable/distributed.html

This command is for the first node in a multi-node setup. Specify the total number of nodes, the rank of the current node (0 for the first node), the master node's IP address, and an available port.

```bash
python -m torch.distributed.launch --nproc-per-node=NUM_GPUS_YOU_HAVE
           --nnodes=2 --node-rank=0 --master-addr="192.168.1.1"
           --master-port=1234 YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3
           and all other arguments of your training script)
```

--------------------------------

### Torch Distributed Elastic (torchrun)

Source: https://docs.pytorch.org/docs/stable/generated/torch.Tensor.gcd_.html

Documentation for PyTorch Distributed Elastic, including quickstart, examples, and details on the `torchrun` utility.

```APIDOC
## Torch Distributed Elastic (torchrun)

### Description
Features and documentation for PyTorch Distributed Elastic, enabling fault-tolerant and elastic distributed training. Includes the `torchrun` command-line utility.

### Key Features
- **Quickstart**: Getting started with elastic distributed training.
- **Train script**: Example training scripts.
- **Examples**: Various usage examples.
- **torchrun (Elastic Launch)**: The command-line launcher for elastic training jobs.
- **Elastic Agent**: The agent responsible for managing worker processes.
- **Multiprocessing**: Details on how multiprocessing is handled.
- **Error Propagation**: How errors are managed and propagated.
- **Rendezvous**: Mechanisms for workers to discover each other.
- **Expiration Timers**: Timeouts for rendezvous and other operations.
- **Metrics**: Collection and reporting of training metrics.
- **Events**: Handling of training events.
- **Subprocess Handling**: Management of worker subprocesses.
- **Control Plane**: The control plane for managing the distributed job.
- **NUMA Binding**: NUMA awareness for performance optimization.
- **Customization**: Options for customizing elastic training behavior.
- **TorchElastic Kubernetes**: Integration with Kubernetes for elastic training.
```

--------------------------------

### Troubleshoot DLL load failure

Source: https://docs.pytorch.org/docs/stable/notes/windows.html

Example of an ImportError caused by missing dependencies, followed by the command to install required MKL components.

```text
from torch._C import *

ImportError: DLL load failed: The specified module could not be found.
```

```bash
pip install numpy mkl intel-openmp mkl_fft
```

--------------------------------

### Initialize and Use FileStore

Source: https://docs.pytorch.org/docs/stable/distributed.html

Illustrates initializing two FileStore instances pointing to the same file, then setting and getting a value.

```python
import torch.distributed as dist
store1 = dist.FileStore("/tmp/filestore", 2)
store2 = dist.FileStore("/tmp/filestore", 2)
# Use any of the store methods from either the client or server after initialization
store1.set("first_key", "first_value")
store2.get("first_key")
```

--------------------------------

### Initialize optimizer and scheduler

Source: https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/torch.compiler_troubleshooting.html

Setup for Adam optimizer and ExponentialLR scheduler.

```python
opt = torch.optim.Adam(mod.parameters(), lr=torch.tensor(0.01))
sched = torch.optim.lr_scheduler.ExponentialLR(opt, torch.tensor(0.9))
```

--------------------------------

### Generate logarithmically spaced tensors

Source: https://docs.pytorch.org/docs/stable/generated/torch.logspace.html

Examples demonstrating the creation of tensors with values spaced on a logarithmic scale using different start, end, steps, and base parameters.

```python
>>> torch.logspace(start=-10, end=10, steps=5)
tensor([ 1.0000e-10,  1.0000e-05,  1.0000e+00,  1.0000e+05,  1.0000e+10])
>>> torch.logspace(start=0.1, end=1.0, steps=5)
tensor([  1.2589,   2.1135,   3.5481,   5.9566,  10.0000])
>>> torch.logspace(start=0.1, end=1.0, steps=1)
tensor([1.2589])
>>> torch.logspace(start=2, end=2, steps=1, base=2)
tensor([4.0])
```

--------------------------------

### Display Launch Utility Help

Source: https://docs.pytorch.org/docs/stable/distributed.html

Run this command to view all available optional arguments and configurations for the torch.distributed.launch utility.

```bash
python -m torch.distributed.launch --help
```

--------------------------------

### Async Execution with TorchScript

Source: https://docs.pytorch.org/docs/stable/rpc.html

This example shows how to combine @rpc.functions.async_execution with @torch.jit.script decorators. When used together, @rpc.functions.async_execution must be the outermost decorator. Ensure RPC setup and shutdown are handled.

```python
from torch import Tensor
from torch.futures import Future
from torch.distributed import rpc

# omitting setup and shutdown RPC

# On all workers
@torch.jit.script
def script_add(x: Tensor, y: Tensor) -> Tensor:
    return x + y

@rpc.functions.async_execution
@torch.jit.script
def async_add(to: str, x: Tensor, y: Tensor) -> Future[Tensor]:
    return rpc.rpc_async(to, script_add, (x, y))

# On worker0
ret = rpc.rpc_sync(
    "worker1",
    async_add,
    args=("worker2", torch.ones(2), 1)
)
print(ret)  # prints tensor([2., 2.])

```

--------------------------------

### Example of Distributed Autograd with FAST mode

Source: https://docs.pytorch.org/docs/stable/rpc/distributed_autograd.html

This example demonstrates setting up and using distributed autograd for a backward pass involving remote computations. Computations must be within the `dist_autograd.context` manager. Ensure RPC is set up before running.

```python
import torch
import torch.distributed.autograd as dist_autograd
import torch.distributed.rpc as rpc

def my_add(t1, t2):
  return torch.add(t1, t2)

# On worker 0:

# Setup the autograd context. Computations that take
# part in the distributed backward pass must be within
# the distributed autograd context manager.
with dist_autograd.context() as context_id:
  t1 = torch.rand((3, 3), requires_grad=True)
  t2 = torch.rand((3, 3), requires_grad=True)

  # Perform some computation remotely.
  t3 = rpc.rpc_sync("worker1", my_add, args=(t1, t2))

  # Perform some computation locally based on remote result.
  t4 = torch.rand((3, 3), requires_grad=True)
  t5 = torch.mul(t3, t4)

  # Compute some loss.
  loss = t5.sum()

  # Run the backward pass.
  dist_autograd.backward(context_id, [loss])

  # Retrieve the gradients from the context.
  dist_autograd.get_gradients(context_id)
```

--------------------------------

### Get Element Size in Bytes for Tensor

Source: https://docs.pytorch.org/docs/stable/generated/torch.Tensor.element_size.html

Returns the size in bytes of an individual element. Use this to understand memory usage per element. Examples show default and uint8 dtypes.

```python
>>> torch.tensor([]).element_size()
4
```

```python
>>> torch.tensor([], dtype=torch.uint8).element_size()
1
```

--------------------------------

### Create a setuptools.Extension for C++

Source: https://docs.pytorch.org/docs/stable/cpp_extension.html

Use CppExtension to create a setuptools.Extension for C++ code. It forwards arguments to the setuptools.Extension constructor. Ensure correct compile and link arguments are provided.

```python
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CppExtension
setup(
    name='extension',
    ext_modules=[
        CppExtension(
            name='extension',
            sources=['extension.cpp'],
            extra_compile_args=['-g'],
            extra_link_args=['-Wl,--no-as-needed', '-lm'])
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
```

--------------------------------

### Custom IterableDataset with Worker Info

Source: https://docs.pytorch.org/docs/stable/data.html

Example of an IterableDataset that splits workload across multiple worker processes using get_worker_info(). Ensure that the start and end values are valid for the dataset range.

```python
class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, start, end):
        super(MyIterableDataset).__init__()
        assert end > start, "this example only works with end >= start"
        self.start = start
        self.end = end

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is None:  # single-process data loading, return the full iterator
            iter_start = self.start
            iter_end = self.end
        else:  # in a worker process
            # split workload
            per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers)))
            worker_id = worker_info.id
```

--------------------------------

### Initialize Modules with Custom Settings

Source: https://docs.pytorch.org/docs/stable/notes/modules.html

Demonstrates initializing modules directly on specific devices, with custom dtypes, or skipping default initialization.

```python
# Initialize module directly onto GPU.
m = nn.Linear(5, 3, device='cuda')

# Initialize module with 16-bit floating point parameters.
m = nn.Linear(5, 3, dtype=torch.half)

# Skip default parameter initialization and perform custom (e.g. orthogonal) initialization.
m = torch.nn.utils.skip_init(nn.Linear, 5, 3)
nn.init.orthogonal_(m.weight)
```

--------------------------------

### Initialize model and state dictionary

Source: https://docs.pytorch.org/docs/stable/distributed.checkpoint.html

Basic setup for creating a model instance and preparing a state dictionary for checkpointing.

```python
>>> my_model = MyModule()
```

```python
>>> state_dict = {"model": my_model}
```

--------------------------------

### Profile ResNet18 with torch.compile

Source: https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/torch.compiler_profiling_torch_compile.html

This example demonstrates how to profile a model compiled with torch.compile. It includes a warm-up run, uses torch.profiler.profile context, and exports the trace to a JSON file. Ensure you have torchvision installed.

```python
import torch
from torchvision.models import resnet18

device = 'cuda'      # or 'cpu', 'xpu', etc.
model = resnet18().to(device)

inputs = [torch.randn((5, 3, 224, 224), device=device) for _ in range(10)]

model_c = torch.compile(model)

def fwd_bwd(inp):
    out = model_c(inp)
    out.sum().backward()

# warm up
fwd_bwd(inputs[0])

with torch.profiler.profile() as prof:
    for i in range(1, 4):
        fwd_bwd(inputs[i])
        prof.step()

prof.export_chrome_trace("trace.json")

```

--------------------------------

### Configure PrepareCustomConfig

Source: https://docs.pytorch.org/docs/stable/generated/torch.ao.quantization.fx.custom_config.PrepareCustomConfig.html

Example usage of the fluent API to set various quantization configurations.

```python
prepare_custom_config = PrepareCustomConfig()             .set_standalone_module_name("module1", qconfig_mapping, example_inputs,                 child_prepare_custom_config, backend_config)             .set_standalone_module_class(MyStandaloneModule, qconfig_mapping, example_inputs,                 child_prepare_custom_config, backend_config)             .set_float_to_observed_mapping(FloatCustomModule, ObservedCustomModule)             .set_non_traceable_module_names(["module2", "module3"])             .set_non_traceable_module_classes([NonTraceableModule1, NonTraceableModule2])             .set_input_quantized_indexes([0])             .set_output_quantized_indexes([0])             .set_preserved_attributes(["attr1", "attr2"])
```

--------------------------------

### Async Execution with Static and Class Methods

Source: https://docs.pytorch.org/docs/stable/rpc.html

This example demonstrates using @rpc.functions.async_execution with static and class methods. When combined, @rpc.functions.async_execution must be the inner decorator. Ensure RPC setup and shutdown are handled.

```python
from torch.distributed import rpc

# omitting setup and shutdown RPC

# On all workers
class AsyncExecutionClass:

    @staticmethod
    @rpc.functions.async_execution
    def static_async_add(to, x, y, z):
        return rpc.rpc_async(to, torch.add, args=(x, y)).then(
            lambda fut: fut.wait() + z
        )

    @classmethod
    @rpc.functions.async_execution
    def class_async_add(cls, to, x, y, z):
        ret_fut = torch.futures.Future()
        rpc.rpc_async(to, torch.add, args=(x, y)).then(
            lambda fut: ret_fut.set_result(fut.wait() + z)
        )
        return ret_fut

    @rpc.functions.async_execution
    def bound_async_add(self, to, x, y, z):
        return rpc.rpc_async(to, torch.add, args=(x, y)).then(
            lambda fut: fut.wait() + z
        )

# On worker0
ret = rpc.rpc_sync(
    "worker1",
    AsyncExecutionClass.static_async_add,
    args=("worker2", torch.ones(2), 1, 2)
)
print(ret)  # prints tensor([4., 4.])

ret = rpc.rpc_sync(
    "worker1",
    AsyncExecutionClass.class_async_add,
    args=("worker2", torch.ones(2), 1, 2)
)
print(ret)  # prints tensor([4., 4.])

```

--------------------------------

### Separate Forward and Context Setup

Source: https://docs.pytorch.org/docs/stable/generated/torch.autograd.Function.forward.html

Use this method when you need to separate the forward pass logic from the context setup. The `forward` method does not accept `ctx`, and context is managed by overriding `setup_context`.

```python
@staticmethod
def forward(*args: Any, **kwargs: Any) -> Any:
    pass


@staticmethod
def setup_context(ctx: Any, inputs: Tuple[Any, ...], output: Any) -> None:
    pass
```

--------------------------------

### Get unique elements from a 2D tensor with inverse indices

Source: https://docs.pytorch.org/docs/stable/generated/torch.functional.unique.html

This example shows how `torch.unique` with `return_inverse=True` operates on a 2D tensor, returning unique scalar elements and their corresponding indices within the flattened tensor.

```python
>>> output, inverse_indices = torch.unique(
...     torch.tensor([[1, 3], [2, 3]], dtype=torch.long), sorted=True, return_inverse=True)
>>> output
tensor([1, 2, 3])
>>> inverse_indices
tensor([[0, 2],
        [1, 2]])
```

--------------------------------

### Configure QConfigMapping with various methods

Source: https://docs.pytorch.org/docs/stable/generated/torch.ao.quantization.qconfig_mapping.QConfigMapping.html

Demonstrates chaining multiple configuration methods to define quantization settings with increasing match priority.

```python
qconfig_mapping = QConfigMapping()
    .set_global(global_qconfig)
    .set_object_type(torch.nn.Linear, qconfig1)
    .set_object_type(torch.nn.ReLU, qconfig1)
    .set_module_name_regex("foo.*bar.*conv[0-9]+", qconfig1)
    .set_module_name_regex("foo.*", qconfig2)
    .set_module_name("module1", qconfig1)
    .set_module_name("module2", qconfig2)
    .set_module_name_object_type_order("foo.bar", torch.nn.functional.linear, 0, qconfig3)
```

--------------------------------

### Registering and Testing Custom Autograd Function in PyTorch

Source: https://docs.pytorch.org/docs/stable/library.html

Register a custom autograd function and test it using `torch.library.opcheck`. This example shows how to define the backward pass and context setup for a custom multiplication operation.

```python
>>>     return grad * ctx.y, None
>>>
>>> numpy_mul.register_autograd(backward, setup_context=setup_context)
>>>
>>> sample_inputs = [
>>>     (torch.randn(3), 3.14),
>>>     (torch.randn(2, 3, device='cuda'), 2.718),
>>>     (torch.randn(1, 10, requires_grad=True), 1.234),
>>>     (torch.randn(64, 64, device='cuda', requires_grad=True), 90.18),
>>> ]
>>>
>>> for args in sample_inputs:
>>>     torch.library.opcheck(numpy_mul, args)
```

--------------------------------

### Register Parameters and Buffers in a Module

Source: https://docs.pytorch.org/docs/stable/notes/modules.html

Comprehensive example showing various ways to register parameters, parameter lists, dictionaries, and buffers.

```python
class StatefulModule(nn.Module):
  def __init__(self):
    super().__init__()
    # Setting a nn.Parameter as an attribute of the module automatically registers the tensor
    # as a parameter of the module.
    self.param1 = nn.Parameter(torch.randn(2))

    # Alternative string-based way to register a parameter.
    self.register_parameter('param2', nn.Parameter(torch.randn(3)))

    # Reserves the "param3" attribute as a parameter, preventing it from being set to anything
    # except a parameter. "None" entries like this will not be present in the module's state_dict.
    self.register_parameter('param3', None)

    # Registers a list of parameters.
    self.param_list = nn.ParameterList([nn.Parameter(torch.randn(2)) for i in range(3)])

    # Registers a dictionary of parameters.
    self.param_dict = nn.ParameterDict({
      'foo': nn.Parameter(torch.randn(3)),
      'bar': nn.Parameter(torch.randn(4))
    })

    # Registers a persistent buffer (one that appears in the module's state_dict).
    self.register_buffer('buffer1', torch.randn(4), persistent=True)

    # Registers a non-persistent buffer (one that does not appear in the module's state_dict).
    self.register_buffer('buffer2', torch.randn(5), persistent=False)

    # Reserves the "buffer3" attribute as a buffer, preventing it from being set to anything
    # except a buffer. "None" entries like this will not be present in the module's state_dict.
    self.register_buffer('buffer3', None)

    # Adding a submodule registers its parameters as parameters of the module.
    self.linear = nn.Linear(2, 3)

m = StatefulModule()

# Save and load state_dict.
torch.save(m.state_dict(), 'state.pt')
m_loaded = StatefulModule()
m_loaded.load_state_dict(torch.load('state.pt'))
```

--------------------------------

### _setup_context

Source: https://docs.pytorch.org/docs/stable/generated/torch.autograd.function.InplaceFunction.html

Defines how to set up the context for the backward pass, either within the forward method or via this dedicated static method.

```APIDOC
## _setup_context(ctx, inputs, output)

### Description
Defines the setup logic for the autograd context. This is used when overriding the forward method signature without context setup inside forward.

### Parameters
- **ctx** (Any) - The context object.
- **inputs** (Any) - The inputs to the forward pass.
- **output** (Any) - The output of the forward pass.
```

--------------------------------

### Async Function with Chained Futures

Source: https://docs.pytorch.org/docs/stable/rpc.html

This example demonstrates using @rpc.functions.async_execution with chained futures. The function returns immediately, allowing concurrent RPC calls, and the result is processed via a callback when the future completes. Ensure RPC setup and shutdown are handled.

```python
from torch.distributed import rpc

# omitting setup and shutdown RPC

# On all workers
@rpc.functions.async_execution
def async_add_chained(to, x, y, z):
    # This function runs on "worker1" and returns immediately when
    # the callback is installed through the `then(cb)` API. In the
    # mean time, the `rpc_async` to "worker2" can run concurrently.
    # When the return value of that `rpc_async` arrives at
    # "worker1", "worker1" will run the lambda function accordingly
    # and set the value for the previously returned `Future`, which
    # will then trigger RPC to send the result back to "worker0".
    return rpc.rpc_async(to, torch.add, args=(x, y)).then(
        lambda fut: fut.wait() + z
    )

# On worker0
ret = rpc.rpc_sync(
    "worker1",
    async_add_chained,
    args=("worker2", torch.ones(2), 1, 1)
)
print(ret)  # prints tensor([3., 3.])

```

--------------------------------

### torch.utils.cpp_extension.CppExtension

Source: https://docs.pytorch.org/docs/stable/cpp_extension.html

Creates a setuptools.Extension for C++ projects with minimal required arguments.

```APIDOC
## CppExtension

### Description
Convenience method that creates a setuptools.Extension with the bare minimum arguments to build a C++ extension. All arguments are forwarded to the setuptools.Extension constructor.

### Parameters
- **name** (str) - Required - The name of the extension.
- **sources** (list) - Required - A list of source filenames.
- **args/kwargs** - Optional - Additional arguments forwarded to setuptools.Extension.
```

--------------------------------

### HingeEmbeddingLoss Example

Source: https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.loss.HingeEmbeddingLoss.html

Example usage of the HingeEmbeddingLoss class.

```APIDOC
## HingeEmbeddingLoss Example

### Code Example
```python
>>> import torch
>>> import torch.nn as nn
>>> loss = nn.HingeEmbeddingLoss()
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.randn(3, 5).sign()
>>> output = loss(input, target)
>>> output.backward()
```
```

--------------------------------

### Creating a model with Sequential

Source: https://docs.pytorch.org/docs/stable/generated/torch.nn.Sequential.html

Demonstrates initializing a Sequential container with a series of layers to create a small model.

```python
model = nn.Sequential(
    nn.Conv2d(1, 20, 5), nn.ReLU(), nn.Conv2d(20, 64, 5), nn.ReLU()
)
```

--------------------------------

### Example Tensor Output

Source: https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/torch.compiler_dynamic_shapes.html

This is an example output tensor from a compiled function.

```python
tensor([  29.8175,  111.1459,  -25.7738,    9.9570,    2.5569,    0.1816,
         -59.7626,  -13.6417,   11.9608,   27.3592,    2.1712,   22.4780,
         -30.7502,   -9.4850,   26.9717,   19.2169,   38.5006,  -57.6737,
          34.3796,   39.4891,   -2.1283,   29.2981,  -40.8374,   -3.7874,
          29.0608,    9.5240,   -8.8026,   48.3119,   58.0510,   51.4383,
          17.6919,  -12.7288,   -5.6804,  -26.4697,   12.2641,   44.3255,
        -104.3630,   32.9295,    4.2355,   28.8946])
```

--------------------------------

### AOTInductor Debugging Guide

Source: https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/torch.compiler_fine_grain_apis.html

Guide for debugging AOTInductor related issues.

```APIDOC
## AOTInductor Debugging Guide

### Description
Provides guidance on debugging problems encountered with AOTInductor.

### Method
N/A (Documentation section)

### Endpoint
N/A

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
N/A

#### Response Example
None
```

--------------------------------

### Implement a model entrypoint

Source: https://docs.pytorch.org/docs/stable/_sources/hub.md.txt

Example implementation of a resnet18 entrypoint including dependencies and docstrings.

```python
  dependencies = ['torch']
  from torchvision.models.resnet import resnet18 as _resnet18

  # resnet18 is the name of entrypoint
  def resnet18(pretrained=False, **kwargs):
      """ # This docstring shows up in hub.help()
      Resnet18 model
      pretrained (bool): kwargs, load pretrained weights into the model
      """
      # Call the model, load pretrained weights
      model = _resnet18(pretrained=pretrained, **kwargs)
      return model
```

--------------------------------

### Register Custom Backend via Python Package Entry Points

Source: https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/torch.compiler_custom_backends.html

Illustrates how to register a custom backend using Python package entry points in `setup.py`. This method is useful for backends distributed as separate packages and allows them to be discovered by name.

```python
...
setup(
    ...
    'torch_dynamo_backends': [
        'my_compiler = your_module.submodule:my_compiler',
    ]
    ...
)


```

--------------------------------

### set_materialize_grads Example

Source: https://docs.pytorch.org/docs/stable/generated/torch.autograd.function.NestedIOFunction.html

Example illustrating how to use `set_materialize_grads` to control gradient materialization.

```APIDOC
### `set_materialize_grads` Example

```python
>>> class SimpleFunc(Function):
>>>     @staticmethod
>>>     def forward(ctx, x):
>>>         return x.clone(), x.clone()
>>>
>>>     @staticmethod
>>>     @once_differentiable
>>>     def backward(ctx, g1, g2):
>>>         return g1 + g2  # No check for None necessary
>>>
>>> # We modify SimpleFunc to handle non-materialized grad outputs
>>> class Func(Function):
>>>     @staticmethod
>>>     def forward(ctx, x):
>>>         ctx.set_materialize_grads(False)
>>>         ctx.save_for_backward(x)
>>>         return x.clone(), x.clone()
>>>
>>>     @staticmethod
>>>     @once_differentiable
>>>     def backward(ctx, g1, g2):
>>>         x, = ctx.saved_tensors
>>>         grad_input = torch.zeros_like(x)
>>>         if g1 is not None:  # We must check for None now
>>>             grad_input += g1
>>>         if g2 is not None:
>>>             grad_input += g2
>>>         return grad_input
>>>
>>> a = torch.tensor(1., requires_grad=True)
>>> b, _ = Func.apply(a)  # induces g2 to be undefined
```
```

--------------------------------

### Prepare a model for quantization aware training using prepare_qat_fx

Source: https://docs.pytorch.org/docs/stable/generated/torch.ao.quantization.quantize_fx.prepare_qat_fx.html

Demonstrates the workflow for preparing a model for QAT, including defining the model, configuring the qconfig_mapping, and executing the preparation function.

```python
import torch
from torch.ao.quantization import get_default_qat_qconfig_mapping
from torch.ao.quantization.quantize_fx import prepare_qat_fx


class Submodule(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = torch.nn.Linear(5, 5)

    def forward(self, x):
        x = self.linear(x)
        return x


class M(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = torch.nn.Linear(5, 5)
        self.sub = Submodule()

    def forward(self, x):
        x = self.linear(x)
        x = self.sub(x) + x
        return x


# initialize a floating point model
float_model = M().train()
# (optional, but preferred) load the weights from pretrained model
# float_model.load_weights(...)


# define the training loop for quantization aware training
def train_loop(model, train_data):
    model.train()
    for image, target in data_loader:
        ...


# qconfig is the configuration for how we insert observers for a particular
# operator
# qconfig = get_default_qconfig("fbgemm")
# Example of customizing qconfig:
# qconfig = torch.ao.quantization.QConfig(
#    activation=FakeQuantize.with_args(observer=MinMaxObserver.with_args(dtype=torch.qint8)),
#    weight=FakeQuantize.with_args(observer=MinMaxObserver.with_args(dtype=torch.qint8)))
# `activation` and `weight` are constructors of observer module

# qconfig_mapping is a collection of quantization configurations, user can
# set the qconfig for each operator (torch op calls, functional calls, module calls)
# in the model through qconfig_mapping
# the following call will get the qconfig_mapping that works best for models
# that target "fbgemm" backend
qconfig_mapping = get_default_qat_qconfig_mapping("fbgemm")

# We can customize qconfig_mapping in different ways, please take a look at
# the docstring for :func:`~torch.ao.quantization.prepare_fx` for different ways
# to configure this

# example_inputs is a tuple of inputs, that is used to infer the type of the
# outputs in the model
# currently it's not used, but please make sure model(*example_inputs) runs
example_inputs = (torch.randn(1, 3, 224, 224),)

# TODO: add backend_config after we split the backend_config for fbgemm and qnnpack
# e.g. backend_config = get_default_backend_config("fbgemm")
# `prepare_qat_fx` inserts observers in the model based on qconfig_mapping and
# backend_config, if the configuration for an operator in qconfig_mapping
# is supported in the backend_config (meaning it's supported by the target
# hardware), we'll insert fake_quantize modules according to the qconfig_mapping
# otherwise the configuration in qconfig_mapping will be ignored
# see :func:`~torch.ao.quantization.prepare_fx` for a detailed explanation of
# how qconfig_mapping interacts with backend_config
prepared_model = prepare_qat_fx(float_model, qconfig_mapping, example_inputs)
# Run training
train_loop(prepared_model, train_loop)
```

--------------------------------

### Example output for logging modes

Source: https://docs.pytorch.org/docs/stable/notes/extending.html

The console output generated by the logging mode example.

```text
TorchFunctionMode logging:
Function Log: torch.rand(*(10,), **{'requires_grad': True})
Function Log: torch.Tensor.mul(*(tensor([0.7164, 0.9897, 0.1745, 0.9336, 0.4287, 0.7989, 0.2169, 0.7474, 0.5624,
        0.5970], requires_grad=True), 2), **None)
Function Log: torch.Tensor.sum(*(tensor([1.4328, 1.9794, 0.3490, 1.8671, 0.8573, 1.5977, 0.4338, 1.4948, 1.1249,
        1.1939], grad_fn=<MulBackward0>),), **None)
# Note that at the python level, we only see the call to backward but not what happens in the autograd engine.
Function Log: torch.Tensor.backward(*(tensor(12.3307, grad_fn=<SumBackward0>),), **{'gradient': None, 'retain_graph': None, 'create_graph': False, 'inputs': None})

TorchDispatchMode logging:
# Here the requires_grad flag from autograd is removed while default arguments were populated.
Dispatch Log: aten.rand.default(*([10],), **{'device': device(type='cpu'), 'pin_memory': False})
Dispatch Log: aten.mul.Tensor(*(tensor([0.2151, 0.6018, 0.8415, 0.9060, 0.2974, 0.7708, 0.6668, 0.0352, 0.7948,
        0.6023], requires_grad=True), 2), **{})
Dispatch Log: aten.sum.default(*(tensor([0.4303, 1.2036, 1.6831, 1.8120, 0.5949, 1.5416, 1.3335, 0.0705, 1.5897,
        1.2046], grad_fn=<MulBackward0>),), **{})
# Here we don't see the call to backward itself, but its constituents. Starting here with the factory function that creates the initial gradient.
Dispatch Log: aten.ones_like.default(*(tensor(11.4637, grad_fn=<SumBackward0>),), **{'pin_memory': False, 'memory_format': torch.preserve_format})
```

--------------------------------

### Example call_function Node

Source: https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/export/ir_spec.html

A specific example of an addition operation node in the FX text format.

```text
%add1 = call_function[target = torch.op.aten.add.Tensor](args = (%x, %y), kwargs = {})
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.