Modular MAX

https://docs.modular.com/
Admin
Modular MAX is an open and fully-integrated suite of AI libraries and tools that accelerates model...
Tokens:862,932
Snippets:7,797
Trust Score:9.9
Update:2 months ago
Show doc for...
Tokens:
Raw
### Initialize and setup a Pixi project

Source: https://docs.modular.com/pixi

Commands to create a new project, enter the directory, and install the Mojo package.

```sh
pixi init example-project
```

```sh
cd example-project
```

```sh
pixi add mojo
```

```sh
pixi run mojo --version
```

--------------------------------

### Run the addition example

Source: https://docs.modular.com/develop/build-custom-ops

Navigate to the custom ops directory and execute the addition example script using pixi.

```sh
cd modular/examples/custom_ops

pixi run python addition.py
```

--------------------------------

### Install max with uv

Source: https://docs.modular.com/max/packages

Install MAX with all dependencies from the nightly index.

```bash
uv add "max[all]" \  
    --index https://whl.modular.com/nightly/simple/ \  
    --prerelease allow  
```

--------------------------------

### Install uv

Source: https://docs.modular.com/max/inference/video-generation

Install the uv package manager.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Install OpenAI SDK

Source: https://docs.modular.com/max/inference/video-generation

Install the OpenAI client using various package managers.

```bash
pixi add openai
```

```bash
uv add openai
```

```bash
pip install openai
```

```bash
conda install openai
```

--------------------------------

### Install MAX with uv

Source: https://docs.modular.com/max/deploy/local-to-cloud

Installs the max package from the nightly index.

```bash
uv add "max[all]" \
    --index https://whl.modular.com/nightly/simple/ \
    --prerelease allow
```

--------------------------------

### Install Pixi

Source: https://docs.modular.com/pixi

Use this command to install Pixi on your system.

```sh
curl -fsSL https://pixi.sh/install.sh | bash
```

--------------------------------

### Clone the Modular examples repository

Source: https://docs.modular.com/gpu-system-profiling

Commands to clone the repository and navigate to the custom operations examples directory.

```bash
git clone https://github.com/modular/modular.git
cd modular/max/examples/custom_ops
```

--------------------------------

### Serve the model

Source: https://docs.modular.com/max/inference/video-generation

Start the local model server.

```bash
max serve \
  --model Wan-AI/Wan2.2-T2V-A14B-Diffusers
```

--------------------------------

### Setup uv virtual environment

Source: https://docs.modular.com/max/inference/video-generation

Create and activate the virtual environment.

```bash
uv venv && source .venv/bin/activate
```

--------------------------------

### Start pixi shell

Source: https://docs.modular.com/max/packages

Activate the project virtual environment.

```bash
pixi shell  
```

--------------------------------

### Start pixi shell

Source: https://docs.modular.com/max/inference/video-generation

Enter the virtual environment.

```bash
pixi shell
```

--------------------------------

### Run the Example

Source: https://docs.modular.com/develop/build-custom-ops

Executes the Python script using the pixi environment manager.

```sh
pixi run python addition.py
```

--------------------------------

### Install dependencies with uv

Source: https://docs.modular.com/max/develop/custom-kernels-pytorch

Install max and required packages using uv with the specified index.

```bash
uv add "max[all]" torch pillow numpy \
    --index https://whl.modular.com/nightly/simple/ \
    --prerelease allow
```

--------------------------------

### Profile a custom operation example

Source: https://docs.modular.com/gpu-system-profiling

Execute a specific example using pixi while capturing profiling data with nsys.

```bash
MODULAR_ENABLE_PROFILING=detailed \
  nsys profile --trace=cuda,osrt,nvtx \
  --cuda-memory-usage=true \
  --output=profile \
  --force-overwrite=true \
  pixi run top_k
```

--------------------------------

### Start model server

Source: https://docs.modular.com/max/deploy/local-to-cloud

Launches the local model server for the specified model.

```bash
max serve --model modularai/Llama-3.1-8B-Instruct-GGUF
```

--------------------------------

### Basic max serve usage

Source: https://docs.modular.com/cli/serve

The standard command syntax for starting the serving process.

```shell
max serve [OPTIONS]
```

--------------------------------

### Add max-all with pixi

Source: https://docs.modular.com/max/packages

Install all MAX dependencies using pixi.

```bash
pixi add max-all  
```

--------------------------------

### Start MAX Container on NVIDIA

Source: https://docs.modular.com/container

Launch the MAX container with NVIDIA GPU support.

```bash
        docker run --gpus=1 \
          -v ~/.cache/huggingface:/root/.cache/huggingface \
          -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
          -p 8000:8000 \
          modular/max-nvidia-full:latest \
          --model google/gemma-3-27b-it
```

--------------------------------

### range(start, stop, step)

Source: https://docs.modular.com/api/python/graph.ops

Generates a sequence of values starting from 'start' up to, but not including, 'stop', with a specified 'step' increment.

```APIDOC
## range(start, stop, step)

### Description
Generates a sequence of values. The sequence starts at the 'start' value and increments by 'step' until it reaches or exceeds the 'stop' value (exclusive).

### Parameters
- **start** (Value | TensorValue | Shape | Dim | HasTensorValue | int | float | integer | floating | DLPackArray) - Required - The first value in the sequence. Must be a scalar value.
- **stop** (Value | TensorValue | Shape | Dim | HasTensorValue | int | float | integer | floating | DLPackArray) - Required - The exclusive upper bound. The sequence stops before this value. Must be a scalar value.
- **step** (Value | TensorValue | Shape | Dim | HasTensorValue | int | float | integer | floating | DLPackArray) - Optional - The spacing between consecutive values. Must be non-zero. Defaults to 1.
```

--------------------------------

### Initialize uv project

Source: https://docs.modular.com/max/packages

Create a new project directory and enter it.

```bash
uv init example-project && cd example-project  
```

--------------------------------

### Initialize uv project

Source: https://docs.modular.com/max/inference/video-generation

Create a new project directory.

```bash
uv init video-generation-quickstart && cd video-generation-quickstart
```

--------------------------------

### Install uv

Source: https://docs.modular.com/max/packages

Command to install the uv package manager.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh  
```

--------------------------------

### Initialize pixi project

Source: https://docs.modular.com/max/inference/video-generation

Create a new project with the required channels.

```bash
pixi init video-generation-quickstart \
  -c https://conda.modular.com/max-nightly/ -c conda-forge \
  && cd video-generation-quickstart
```

--------------------------------

### Initialize uv project

Source: https://docs.modular.com/max/inference/image-to-text

Create a new project directory using uv.

```bash
uv init vision-quickstart && cd vision-quickstart
```

--------------------------------

### Install pixi

Source: https://docs.modular.com/max/packages

Command to install the pixi package manager.

```bash
curl -fsSL https://pixi.sh/install.sh | sh  
```

--------------------------------

### Initialize pixi project

Source: https://docs.modular.com/max/inference/image-to-text

Create a new project directory with the required channels.

```bash
pixi init vision-quickstart \
  -c https://conda.modular.com/max-nightly/ -c conda-forge \
  && cd vision-quickstart
```

--------------------------------

### Initialize uv project

Source: https://docs.modular.com/max/develop/custom-kernels-pytorch

Create a new project directory and initialize it using uv.

```bash
uv init pytorch_custom_ops && cd pytorch_custom_ops
```

--------------------------------

### Install numactl

Source: https://docs.modular.com/max/gpu-system-profiling

Commands to install the numactl package on common Linux distributions.

```bash
# Ubuntu/Debian  

sudo apt-get install numactl  

  

# RHEL/CentOS/Fedora  

sudo dnf install numactl
```

--------------------------------

### Prepare Azure Startup Script

Source: https://docs.modular.com/max/deploy/local-to-cloud

Define and base64 encode the startup script for the VM.

```bash
STARTUP_SCRIPT='#!/bin/bash  

  

sudo usermod -aG docker $USER  

  

sudo systemctl restart docker  

  

sleep 10  

  

HF_TOKEN=$1  

HUGGING_FACE_REPO_ID=${2:-modularai/Llama-3.1-8B-Instruct-GGUF}  

  

sudo docker run -d \

  --restart unless-stopped \

  --env "HF_TOKEN=${HF_TOKEN}" \

  -v ~/.cache/huggingface:/root/.cache/huggingface \

  -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \

  --gpus 1 \

  -p 80:8000 \

  --ipc=host \

  modular/max-nvidia-full:latest \

  --model ${HUGGING_FACE_REPO_ID}'  

  

export STARTUP_SCRIPT=$(echo "$STARTUP_SCRIPT" | base64)  


```

--------------------------------

### Install numactl on Linux

Source: https://docs.modular.com/gpu-system-profiling

Commands to install the numactl package on common Linux distributions.

```bash
# Ubuntu/Debian
sudo apt-get install numactl

# RHEL/CentOS/Fedora
sudo dnf install numactl
```

--------------------------------

### Install OpenAI Python API

Source: https://docs.modular.com/container

Install the necessary library to interact with the MAX endpoint.

```bash
pip install openai
```

--------------------------------

### Initialize uv project

Source: https://docs.modular.com/max/deploy/local-to-cloud

Creates a new project directory and initializes it using uv.

```bash
uv init llama3-tutorial && cd llama3-tutorial
```

--------------------------------

### Start and End Trace Recording

Source: https://docs.modular.com/api/mojo/max/runtime/tracing/Trace

Manual methods to start and end trace event recording.

```mojo
def start(mut self)
```

```mojo
def end(mut self)
```

--------------------------------

### Prepare Azure Startup Script

Source: https://docs.modular.com/serve/local-to-cloud

Creates and base64 encodes the startup script for the Azure VM.

```bash
STARTUP_SCRIPT='#!/bin/bash

sudo usermod -aG docker $USER

sudo systemctl restart docker

sleep 10

HF_TOKEN=$1
HUGGING_FACE_REPO_ID=${2:-modularai/Llama-3.1-8B-Instruct-GGUF}

sudo docker run -d \
  --restart unless-stopped \
  --env "HF_TOKEN=${HF_TOKEN}" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
  --gpus 1 \
  -p 80:8000 \
  --ipc=host \
  modular/max-nvidia-full:latest \
  --model ${HUGGING_FACE_REPO_ID}'

export STARTUP_SCRIPT=$(echo "$STARTUP_SCRIPT" | base64)
```

--------------------------------

### Initialize pixi project

Source: https://docs.modular.com/max/packages

Create a new project with specific channels for MAX nightly builds.

```bash
pixi init example-project \  
  -c https://conda.modular.com/max-nightly/ -c conda-forge \  
  && cd example-project  
```

--------------------------------

### Install OpenAI Package

Source: https://docs.modular.com/get-started

Install the OpenAI client library using various package managers.

```bash
pixi add openai
```

```bash
uv add openai
```

```bash
pip install openai
```

```bash
conda install -c conda-forge openai
```

--------------------------------

### Create and use DeviceRef

Source: https://docs.modular.com/api/python/generated/max.graph.DeviceKind

Demonstrates initializing GPU and CPU device references with specific IDs.

```python
from max.graph import DeviceRef
gpu_device = DeviceRef.GPU()
print(gpu_device)  # Outputs: gpu:0
# Create a CPU device with specific id
cpu_device = DeviceRef.CPU(id=1)
print(cpu_device)  # Outputs: cpu:1
```

--------------------------------

### Install specific MAX skill with npx

Source: https://docs.modular.com/develop/model-bringup-skills

Installs only the import-model skill using the --skill flag.

```bash
npx skills add modular/skills --skill import-model
```

--------------------------------

### Initialize pixi project

Source: https://docs.modular.com/max/deploy/local-to-cloud

Creates a new project directory and initializes it with the required Modular channels.

```bash
pixi init llama3-tutorial \
  -c https://conda.modular.com/max-nightly/ -c conda-forge \
  && cd llama3-tutorial
```

--------------------------------

### Install pixi environment manager

Source: https://docs.modular.com/develop/build-custom-ops

Install the pixi tool to manage dependencies and virtual environments for the project.

```sh
curl -fsSL https://pixi.sh/install.sh | sh
```

--------------------------------

### Initialize Google Cloud SDK

Source: https://docs.modular.com/max/deploy/local-to-cloud

Commands for initializing and authenticating with the Google Cloud SDK.

```bash
gcloud init
```

```bash
gcloud auth login
```

--------------------------------

### Install Nsight Systems on Ubuntu/Debian

Source: https://docs.modular.com/gpu-system-profiling

Use the package manager to install Nsight Systems on Debian-based Linux distributions.

```bash
sudo apt-get install nsight-systems
```

--------------------------------

### Install Modular AI agent skills

Source: https://docs.modular.com/coding-assistants

Use this command to install official AI agent skills for the Modular Platform.

```sh
npx skills add modular/skills
```

--------------------------------

### Configure and Start the MAX Model Server

Source: https://docs.modular.com/serve/video-generation

Set the required environment variable to enable the responses endpoint and start the server with the specified model.

```bash
export MAX_SERVE_API_TYPES='["responses"]'
```

```bash
max serve \
  --model Wan-AI/Wan2.2-T2V-A14B-Diffusers
```

--------------------------------

### Tensor.device

Source: https://docs.modular.com/api/python/generated/max.experimental.tensor.Tensor

Gets the device where the tensor is stored.

```APIDOC
## property device

### Description
Returns the device (CPU or accelerator) where the tensor’s data is located. Raises an error for distributed tensors that span multiple devices.

### Returns
- **Device** - The device where the tensor is stored.
```

--------------------------------

### __len__

Source: https://docs.modular.com/api/mojo/layout/runtime_layout/RuntimeLayout

Gets the number of dimensions in the layout.

```APIDOC
## __len__

### Description
Get the number of dimensions in the layout.

### Returns
- **Int** - The number of dimensions (rank) of the layout.
```

--------------------------------

### Initialize Mojo Package

Source: https://docs.modular.com/develop/custom-kernels-pytorch

Create an empty __init__.mojo file to mark the operations directory as a valid Mojo package.

```bash
touch operations/__init__.mojo
```

--------------------------------

### slice_as_view[dtype, start_type, end_type, step_type](tensor, starts, ends, steps)

Source: https://docs.modular.com/api/mojo/nn/slice/slice_as_view

Returns a zero-copy view of a TileTensor sliced along every dimension. The function normalizes and clamps the provided start, end, and step indices for each axis.

```APIDOC
## slice_as_view

### Description
Returns a view of `tensor` sliced along every dimension. For each axis, the corresponding entries in `starts`, `ends`, and `steps` are normalized and clamped, then the offset, stride, and extent of that dimension are adjusted to produce a zero-copy view of the source data.

### Signature
`def slice_as_view[dtype: DType, start_type: DType, end_type: DType, step_type: DType](tensor: TileTensor, starts: TileTensor, ends: TileTensor, steps: TileTensor) -> TileTensor`

### Arguments
- **tensor** (TileTensor) - Source tensor to slice.
- **starts** (TileTensor) - One-dimensional tensor of starting indices, one per rank.
- **ends** (TileTensor) - One-dimensional tensor of stopping indices (exclusive), one per rank.
- **steps** (TileTensor) - One-dimensional tensor of strides, one per rank; each must be non-zero.

### Returns
- **TileTensor** - A zero-copy view of the source tensor with adjusted dimensions.
```

--------------------------------

### Example model configuration structure

Source: https://docs.modular.com/develop/read-model-config

A sample JSON-like configuration output showing key architectural parameters such as architectures, model_type, and torch_dtype.

```json
LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "torch_dtype": "bfloat16",
  "eos_token_id": 2,
  "head_dim": 64,
  "hidden_act": "silu",
  "hidden_size": 2048,
  "initializer_range": 0.02,
  "intermediate_size": 5632,
  "max_position_embeddings": 2048,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 22,
  "num_key_value_heads": 4,
  "pad_token_id": null,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_parameters": {
    "rope_theta": 10000.0,
    "rope_type": "default"
  },
  "tie_word_embeddings": false,
  "transformers_version": "5.9.0",
  "use_cache": true,
  "vocab_size": 32000
}
```

--------------------------------

### gpu_qint4_repack_Q4_0

Source: https://docs.modular.com/api/mojo/quantization/qmatmul_gpu/gpu_qint4_repack_Q4_0

Launches the GPU kernel that repacks Q4_0 weights into the packed GEMM layout.

```APIDOC
## gpu_qint4_repack_Q4_0

### Description
Launches the GPU kernel that repacks Q4_0 weights into the packed GEMM layout.

### Signature
`def gpu_qint4_repack_Q4_0[target: StringSlice[ImmStaticOrigin]](b_tt: TileTensor[DType.uint8, Storage=b_tt.Storage, linear_idx_type=b_tt.linear_idx_type], b_packed_tt: TileTensor[DType.uint8, Storage=b_packed_tt.Storage, linear_idx_type=b_packed_tt.linear_idx_type], ctx: Optional[DeviceContext] = None)`

### Parameters
- **target** (StringSlice[ImmStaticOrigin]) - Required - The target platform string, which must identify a GPU.
- **b_tt** (TileTensor[DType.uint8]) - Required - The input Q4_0 quantized weight tile tensor in global memory.
- **b_packed_tt** (TileTensor[DType.uint8]) - Required - The output repacked weight tile tensor in global memory.
- **ctx** (Optional[DeviceContext]) - Optional - The device context used to enqueue the kernel.

### Raises
- An error if the input tensors are not rank-2 or the target is not a GPU.
```

--------------------------------

### gpu_qint4_repack_Q4_0

Source: https://docs.modular.com/api/mojo/quantization/qmatmul_gpu

Launches the GPU kernel that repacks Q4_0 weights into the packed GEMM layout.

```APIDOC
## gpu_qint4_repack_Q4_0

### Description
Launches the GPU kernel that repacks GGUF Q4_0 weights into the packed GEMM layout required for quantized matrix multiplication.
```

--------------------------------

### Add modular to pixi

Source: https://docs.modular.com/max/inference/video-generation

Install the modular package.

```bash
pixi add modular
```

--------------------------------

### Start MAX Model Server

Source: https://docs.modular.com/get-started

Launch the model server using the MAX CLI with specific model configurations.

```sh
max serve --model {image} --trust-remote-code
```

```sh
max serve --model {video}
```

--------------------------------

### Inference Output

Source: https://docs.modular.com/container

Example output returned by the model.

```output
The **Los Angeles Dodgers** won the World Series in 2020!

They defeated the Tampa Bay Rays 4 games to 2. It was their first World Series title since 1988.

It was a unique World Series as it was played in a neutral site (Globe Life Field in Arlington, Texas) due to the COVID-19 pandemic.
```

--------------------------------

### dim

Source: https://docs.modular.com/api/mojo/layout/runtime_layout/RuntimeLayout

Gets the size of the dimension at the specified index.

```APIDOC
## dim

### Description
Get the size of the dimension at the specified index.

### Parameters
- **i** (Int) - Required - The index of the dimension to retrieve.

### Returns
- **Int** - The size of the dimension at index i.
```

--------------------------------

### Start a model server

Source: https://docs.modular.com/cli/serve

Launches a server for a specific model on a designated GPU with defined batch size and memory utilization.

```bash
max serve \
  --model google/gemma-3-12b-it \
  --devices gpu:0 \
  --max-batch-size 8 \
  --device-memory-utilization 0.9
```

--------------------------------

### GET /v1/models

Source: https://docs.modular.com/max/rest-api

List the models available in the MAX API.

```APIDOC
## GET /v1/models

### Description
List the models available in the MAX API.

### Method
GET

### Endpoint
/v1/models
```

--------------------------------

### Create and activate uv virtual environment

Source: https://docs.modular.com/max/packages

Set up a virtual environment and activate it in the current shell.

```bash
uv venv && source .venv/bin/activate  
```

--------------------------------

### Start a local model server

Source: https://docs.modular.com/serve/text-to-text

Use the max serve command to host a model on a local endpoint.

```bash
max serve --model google/gemma-4-31B-it
```

--------------------------------

### get_type_name

Source: https://docs.modular.com/api/mojo/nn/attention/mha_mask/MHAMask

Gets the name of the host type implementing the trait.

```APIDOC
## get_type_name()

### Description
Returns the string representation of the host type, used primarily for error messaging.

### Returns
- **String** - The host type name.
```

--------------------------------

### get

Source: https://docs.modular.com/api/mojo/layout/layout_tensor/LayoutTensorIter

Retrieves the layout tensor at the current iterator position.

```APIDOC
## get

### Description
Returns a layout tensor representing the data at the current position of the iterator.

### Returns
- **LayoutTensorType** - A tensor view at the current iterator position.
```

--------------------------------

### Trace.start / Trace.__enter__

Source: https://docs.modular.com/api/mojo/max/runtime/tracing/Trace

Begins recording a trace event.

```APIDOC
## Trace.start / Trace.__enter__

### Description
Starts the recording of a trace event. `__enter__` is used for context management, while `start` is an explicit method call.

### Raises
- Raises an error if the operation fails.
```

--------------------------------

### max.graph.ops.range

Source: https://docs.modular.com/api/python/graph.ops

Creates a sequence of evenly spaced values from start to stop.

```APIDOC
## max.graph.ops.range(start, stop, step=1, out_dim=None, *, dtype, device)

### Description
Creates a sequence of evenly spaced values from start to stop. The sequence begins at start and increments by step, stopping before stop.

### Parameters
- **start** (Any) - Required - The start value of the sequence.
- **stop** (Any) - Required - The end value of the sequence (exclusive).
- **step** (Any) - Optional - The increment value (default: 1).
- **out_dim** (Any) - Optional - The output dimension.
- **dtype** (DType) - Required - The data type of the output tensor.
- **device** (DeviceRef) - Required - The device on which to create the sequence.
```
Tokens:
Raw