### Install clang-format (Ubuntu Example)

Source: https://github.com/kvcache-ai/mooncake/blob/main/CONTRIBUTING.md

Install the clang-format tool, which is used for C/C++ code formatting, on Ubuntu systems.

```bash
sudo apt-get update && sudo apt-get install -y clang-format-20
```

--------------------------------

### Start Proxy Server for Disaggregated Example

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/vllm-integration/kv-cache-storage.md

This command starts the proxy server for the disaggregated vLLM example. It requires specifying the model, ports for prefill and decode instances, and the proxy server's listening port. Ensure you are in the 'vllm' directory.

```bash
# 5. Start the proxy server
cd vllm
python3 examples/online_serving/disagg_examples/disagg_proxy_demo.py \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --prefill localhost:8100 localhost:8101 \
    --decode localhost:8200 localhost:8201 \
    --port 8000
```

--------------------------------

### Run Example: vLLM with Mooncake KV Cache

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/vllm-integration/kv-cache-storage.md

This section details the commands to start the necessary services for vLLM integration with Mooncake's KV cache. Ensure you start from the root of your cloned repository. Note that `VLLM_USE_V1=0` is required for disaggregated features.

```bash
# Begin from root of your cloned repo!

# 1. Start the etcd server
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379
# You may need to terminate other etcd processes before running the above command

# 2. Start the mooncake_master server
mooncake_master --port 50001
# If some vllm instances exit unexpectedly, some connection metadata will be
# corrupted since they are not properly cleaned. In that case, we recommend
# you restart the mooncake_master before running another test.

# 3. Run multiple vllm instances
# kv_producer role
MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8100 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'

CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8101 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'

CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8102 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'

CUDA_VISIBLE_DEVICES=3 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8103 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'

# kv_consumer role
CUDA_VISIBLE_DEVICES=4 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8200 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'

CUDA_VISIBLE_DEVICES=5 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8201 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'

CUDA_VISIBLE_DEVICES=6 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8102 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'

CUDA_VISIBLE_DEVICES=7 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
    --port 8203 \
    --max-model-len 10000 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
```

--------------------------------

### Launch Toy Proxy Server for Integration

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/performance/vllm-v1-support-benchmark.md

Starts a toy proxy server to facilitate integration between the prefiller and decoder nodes. This is essential for routing requests in the distributed setup.

```bash
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
  --host 0.0.0.0 --port 8000 \
  --prefiller-host 10.0.28.193 --prefiller-port 8010 \
  --decoder-host 10.0.28.202 --decoder-port 8020
```

--------------------------------

### Start HTTP Metadata Server

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/transfer-engine/index.md

Runs the HTTP metadata server example. This server is used for centralized metadata management for Transfer Engine.

```bash
# cd mooncake-transfer-engine/example/http-metadata-server
go run . --addr=:8080
```

--------------------------------

### MooncakeDistributedStore Setup and Usage

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Demonstrates how to initialize, set up, and use the MooncakeDistributedStore to store and retrieve data. This example includes setting up with an HTTP metadata server.

```APIDOC
## MooncakeDistributedStore Setup and Usage

### Description
This example shows how to create an instance of `MooncakeDistributedStore`, configure it with necessary parameters including an HTTP metadata server, store a key-value pair, retrieve the value, and finally close the store connection.

### Method
```python
from mooncake.store import MooncakeDistributedStore

# 1. Create store instance
store = MooncakeDistributedStore()

# 2. Setup with all required parameters
store.setup(
    "localhost",           # Your node's address
    "http://localhost:8080/metadata",    # HTTP metadata server
    512*1024*1024,          # 512MB segment size
    128*1024*1024,          # 128MB local buffer
    "tcp",                             # Use TCP (RDMA for high performance)
    "",                            # Leave empty; Mooncake auto-picks RDMA devices when needed
    "localhost:50051"        # Master service
)

# 3. Store data
store.put("hello_key", b"Hello, Mooncake Store!")

# 4. Retrieve data
data = store.get("hello_key")
print(data.decode())  # Output: Hello, Mooncake Store!

# 5. Clean up
store.close()
```

### Parameters for `setup()`

- **node_address** (string) - Required - The address of your node.
- **metadata_server** (string) - Required - The URL of the HTTP metadata server or "P2PHANDSHAKE" for peer-to-peer discovery.
- **segment_size** (int) - Required - The size of data segments in bytes.
- **local_buffer** (int) - Required - The size of the local buffer in bytes.
- **protocol** (string) - Required - The network protocol to use (e.g., "tcp", "rdma", "efa").
- **rdma_devices** (string) - Optional - Comma-separated list of RDMA devices to use. Leave empty for auto-discovery.
- **master_service** (string) - Required - The address of the master service (e.g., "host:port").

### Methods

- **`put(key: str, value: bytes)`**: Stores a key-value pair in the store.
- **`get(key: str)`**: Retrieves the value associated with a given key.
- **`close()`**: Closes the store connection and cleans up resources.
```

--------------------------------

### P2P Hello World Example with Mooncake Store

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

An example using P2P handshake for Mooncake Store setup, which does not require an HTTP metadata server. The literal 'P2PHANDSHAKE' is passed as the metadata server address.

```python
from mooncake.store import MooncakeDistributedStore

store = MooncakeDistributedStore()
store.setup(
    "localhost",           # Your node's ip address
    "P2PHANDSHAKE",              # P2P handshake (no HTTP metadata)
    512*1024*1024,                # 512MB segment size
    128*1024*1024,                # 128MB local buffer
    "tcp",                       # Use TCP (RDMA for high performance)
    "",                          # Leave empty; Mooncake auto-picks RDMA devices when needed
    "localhost:50051"           # Master service
)

store.put("hello_key", b"Hello, Mooncake Store!")
print(store.get("hello_key").decode())
store.close()
```

--------------------------------

### Launch Router for Multi-Node with Multiple Decode Instances

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/sglang-integration-v1.md

This example shows how to configure the router for a multi-node setup where multiple decode nodes are running on different hosts. The router will distribute requests across these decode instances.

```bash
# Router for multi node.
# Here is an example for 2 decode node running on 192.168.0.137 and 192.168.0.140
python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill "http://192.168.0.137:30000" 8998 \
  --decode "http://192.168.0.137:30001,http://192.168.0.140:30001" \
  --policy round_robin \
  --host 0.0.0.0 --port 8000
```

--------------------------------

### Quick Start: Initialize and Use Mooncake Store in Rust

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/api-reference/rust/mooncake-store.md

Demonstrates basic usage of the `mooncake_store` crate, including initialization, setup, putting, getting, and removing key-value pairs. Ensure the Mooncake Store C++ library is built and accessible at runtime.

```rust
use mooncake_store::MooncakeStore;

fn main() -> Result<(), mooncake_store::StoreError> {
    let store = MooncakeStore::new()?;
    store.setup(
        "127.0.0.1",
        "http://127.0.0.1:8080/metadata",
        512 << 20, // global_segment_size
        128 << 20, // local_buffer_size
        "tcp",
        "",
        "127.0.0.1:50051",
    )?;

    store.put("hello", b"world", None)?;
    let value = store.get("hello")?;
    assert_eq!(value, b"world");

    store.remove("hello", false)?;
    Ok(())
}
```

--------------------------------

### Install vLLM from Source

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/vllm-integration/kv-cache-storage.md

Installs the vLLM package in editable mode after cloning the repository. Refer to vLLM's official guide for troubleshooting.

```bash
cd vllm
pip3 install -e .
```

--------------------------------

### Build LMDeploy from Source

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/lmdeploy-integration-v0.9.md

Navigate to the LMDeploy directory and install it in editable mode. Refer to the official guide for troubleshooting.

```bash
cd LMDeploy
pip install -e .
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/README.md

Install the necessary Python packages required for building the documentation. Ensure you are in the 'docs' directory.

```bash
pip install -r ../requirements_docs.txt
```

--------------------------------

### Start Mooncake Client

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/performance/ssd-offload-benchmark-results.md

Launches the Mooncake client with specified configurations for host, segment size, master server, metadata server, protocol, and devices. Ensure the protocol and device names match your network setup.

```bash
mooncake_client \
    --host=127.0.0.1 \
    --global_segment_size=80GB \
    --master_server_address=localhost:50051 \
    --metadata_server=P2PHANDSHAKE \
    --protocol=rdma \
    --device_names=ibp12s0,ibp75s0 \
    --port=50052 \
    --logtostderr
```

--------------------------------

### Install sglang-router Package

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/sglang-integration/hicache-quick-start.md

Install the router package to provide the sglang_router entrypoint. This is a prerequisite for launching SGLang servers.

```bash
pip install sglang-router
```

--------------------------------

### Install UBSHMEM Allocator SO

Source: https://github.com/kvcache-ai/mooncake/blob/main/mooncake-integration/CMakeLists.txt

Installs the ubshmem_fabric_allocator.so file to the Python package destination if USE_UBSHMEM is enabled.

```cmake
if(USE_UBSHMEM)
  message(
    STATUS
      "USE_UBSHMEM is enabled, ubshmem_fabric_allocator.so will be installed in the Python package"
  )
  install(FILES "${UBSHMEM_ALLOCATOR_SO_PATH}"
          DESTINATION ${PYTHON_SYS_PATH}/${PYTHON_PACKAGE_NAME})
endif()
```

--------------------------------

### Install libclang-dev and clang on Ubuntu

Source: https://github.com/kvcache-ai/mooncake/blob/main/mooncake-store/rust/README.md

Install the necessary prerequisites for building the Rust crate. This command is specific to Ubuntu systems.

```bash
sudo apt-get install libclang-dev clang
```

--------------------------------

### Install Async Store Python Script

Source: https://github.com/kvcache-ai/mooncake/blob/main/mooncake-integration/CMakeLists.txt

Installs the async_store.py script to the Python package destination.

```cmake
install(FILES "${CMAKE_CURRENT_SOURCE_DIR}/store/async_store.py"
        DESTINATION ${PYTHON_SYS_PATH}/${PYTHON_PACKAGE_NAME})
```

--------------------------------

### Install aiohttp Library

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/deployment/nvmf-ssd-deployment-guide.md

Installs the aiohttp library using pip3, necessary for the HTTP metadata server if errors occur during startup.

```bash
pip3 install aiohttp
```

--------------------------------

### Install Development Dependencies and Pre-commit Hooks

Source: https://github.com/kvcache-ai/mooncake/blob/main/CONTRIBUTING.md

Install necessary development packages and set up pre-commit hooks for consistent code formatting and checks.

```bash
pip install -r requirements-dev.txt
pre-commit install
```

--------------------------------

### Allocate and Mount Memory Example (curl)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/http-api-reference/http-service.md

Example using curl to POST a request to the /api/mount endpoint to allocate and mount a memory segment.

```bash
curl -X POST http://localhost:8080/api/mount \
  -H "Content-Type: application/json" \
  -d '{"size": 16777216, "protocol": "tcp", "location": ""}'
```

--------------------------------

### setup()

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Initializes distributed resources and establishes network connections for the Mooncake Store. This method requires essential connection details and allows configuration of network protocols, buffer sizes, and offload settings.

```APIDOC
## setup()

### Description
Initialize distributed resources and establish network connections.

### Method
```python
def setup(
    self,
    local_hostname: str,
    metadata_server: str,
    global_segment_size: int = 16777216,
    local_buffer_size: int = 1073741824,
    protocol: str = "tcp",
    rdma_devices: str = "",
    master_server_addr: str,
    engine: Optional[TransferEngine] = None,
    enable_ssd_offload: bool = False,
    ssd_offload_path: str = "",
    tenant_id: str = "default",
) -> int
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- `local_hostname` (str): **Required**. Local hostname and port (e.g., “localhost” or “localhost:12345”)
- `metadata_server` (str): **Required**. Metadata connection string, e.g. `"P2PHANDSHAKE"` or `"http://localhost:8080/metadata"`.
- `global_segment_size` (int): Memory segment size in bytes for mounting.
- `local_buffer_size` (int): Local buffer size in bytes.
- `protocol` (str): Network protocol, usually `"tcp"`, `"rdma"`, `"efa"`, `"cxl"`, or `"ascend"` depending on the build.
- `rdma_devices` (str): RDMA/EFA device name(s), e.g. `"mlx5_0"` or `"mlx5_0,mlx5_1"`. Leave empty to auto-discover NICs unless `MC_MS_AUTO_DISC=0`; always empty for TCP.
- `master_server_addr` (str): **Required**. Master server address (e.g., “localhost:50051”)
- `engine` (Optional[TransferEngine]): Existing Transfer Engine instance to reuse. Defaults to `None`.
- `enable_ssd_offload` (bool): Enable client-side SSD offload support. Defaults to `False`.
- `ssd_offload_path` (str): SSD offload directory. When provided, overrides the storage path environment configuration.
- `tenant_id` (str): Tenant namespace for object keys. Defaults to `"default"`.

### Returns
- `int`: Status code (0 = success, non-zero = error code)

### Request Example
```python
# TCP initialization
store.setup("localhost", "http://localhost:8080/metadata", 1024*1024*1024, 128*1024*1024, "tcp", "", "localhost:50051")

# RDMA auto-detect
store.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 128*1024*1024, "rdma", "", "localhost:50051")

# RDMA with explicit device list
store.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 128*1024*1024, "rdma", "mlx5_0,mlx5_1", "localhost:50051")
```
```

--------------------------------

### Install ASIO Shared Library

Source: https://github.com/kvcache-ai/mooncake/blob/main/mooncake-common/src/CMakeLists.txt

Installs the ASIO shared library to the 'lib' directory. This makes the ASIO implementation library available for use.

```cmake
install(TARGETS asio_shared DESTINATION lib)
```

--------------------------------

### Hello World Example for Mooncake Store

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/quick-start.md

A basic 'Hello World' example demonstrating how to initialize the MooncakeDistributedStore, set it up with necessary parameters, store a key-value pair, retrieve it, and close the store.

```python
from mooncake.store import MooncakeDistributedStore

# 1. Create store instance
store = MooncakeDistributedStore()

# 2. Setup with all required parameters
store.setup(
    "localhost",           # Your node's address
    "http://localhost:8080/metadata",    # HTTP metadata server
    512*1024*1024,          # 512MB segment size
    128*1024*1024,          # 128MB local buffer
    "tcp",                  # Use TCP (RDMA for high performance)
    "",                      # Leave empty; Mooncake auto-picks RDMA devices when needed
    "localhost:50051"        # Master service
)

# 3. Store data
store.put("hello_key", b"Hello, Mooncake Store!")

# 4. Retrieve data
data = store.get("hello_key")
print(data.decode())  # Output: Hello, Mooncake Store!

# 5. Clean up
store.close()
```

--------------------------------

### MooncakeStore Quick Start

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/api-reference/rust/mooncake-store.md

Demonstrates how to initialize and use the `MooncakeStore` for basic operations. This includes setting up the store connection, putting a key-value pair, retrieving a value, and removing a key.

```APIDOC
## MooncakeStore Operations

### Description
This section provides a practical example of how to use the `MooncakeStore` Rust API for common operations.

### Usage
```rust
use mooncake_store::MooncakeStore;

fn main() -> Result<(), mooncake_store::StoreError> {
    // Initialize a new MooncakeStore instance.
    let store = MooncakeStore::new()?;

    // Setup the store connection with necessary parameters.
    store.setup(
        "127.0.0.1",                     // host
        "http://127.0.0.1:8080/metadata", // metadata_url
        512 << 20,                       // global_segment_size
        128 << 20,                       // local_buffer_size
        "tcp",                           // protocol
        "",                              // auth_token (empty for no auth)
        "127.0.0.1:50051",               // endpoint
    )?;

    // Put a key-value pair into the store.
    store.put("hello", b"world", None)?;

    // Get the value associated with a key.
    let value = store.get("hello")?;
    assert_eq!(value, b"world");

    // Remove a key from the store.
    store.remove("hello", false)?;

    Ok(())
}
```

### Methods
- **`MooncakeStore::new()`**: Creates a new instance of `MooncakeStore`.
- **`MooncakeStore::setup(...)`**: Configures and establishes the connection to the Mooncake Store.
- **`MooncakeStore::put(key: &str, value: &[u8], ttl: Option<u64>)`**: Inserts or updates a key-value pair. `ttl` is an optional time-to-live in seconds.
- **`MooncakeStore::get(key: &str)`**: Retrieves the value associated with a given key.
- **`MooncakeStore::remove(key: &str, force: bool)`**: Removes a key. `force` indicates whether to force removal.

### Errors
Operations can return `mooncake_store::StoreError` on failure.
```

--------------------------------

### Start Mooncake Store Service (Console Script)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/http-api-reference/http-service.md

Starts the Mooncake store REST service using the installed console script. Requires a configuration file and a port number.

```bash
mc_store_rest_server --config /path/to/mooncake_config.json --port 8080
```

--------------------------------

### Core Usage Path Example (C++)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/tent/cpp-api.md

Demonstrates the typical workflow for using the TransferEngine: creating the engine, registering memory, opening a segment, submitting a transfer, polling for status, and cleaning up resources. Ensure the engine is available before proceeding.

```cpp
#include "tent/transfer_engine.h"

using mooncake::tent::TransferEngine;
using mooncake::tent::Request;
using mooncake::tent::TransferStatus;
using mooncake::tent::SegmentID;
using mooncake::tent::BatchID;

// Create engine (loads config from default path or environment)
TransferEngine engine;
if (!engine.available()) {
    // handle initialization failure
}

// Allocate and register local memory
void* local_addr = nullptr;
size_t length = 1024 * 1024;  // 1MB
engine.allocateLocalMemory(&local_addr, length, "cuda:0");

// Open remote segment
SegmentID remote_segment;
engine.openSegment(remote_segment, "remote_node");

// Prepare transfer request
Request req{};
req.opcode = Request::WRITE;
req.source = local_addr;
req.target_id = remote_segment;
req.target_offset = 0;
req.length = length;

// Allocate batch and submit transfer
BatchID batch = engine.allocateBatch(1);
engine.submitTransfer(batch, {req});

// Poll for completion
TransferStatus status;
do {
    engine.getTransferStatus(batch, status);
} while (status.s == mooncake::tent::PENDING);

// Cleanup
engine.freeBatch(batch);
engine.closeSegment(remote_segment);
engine.freeLocalMemory(local_addr);

```

--------------------------------

### setup_dummy()

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Initializes the Mooncake Store with a dummy client, suitable for testing purposes. It requires specifying memory pool size, local buffer size, and the server address.

```APIDOC
## setup_dummy()

### Description
Initialize the store with a dummy client for testing purposes.

### Method
```python
def setup_dummy(self, mem_pool_size: int, local_buffer_size: int, server_address: str) -> int
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- `mem_pool_size` (int): Memory pool size in bytes
- `local_buffer_size` (int): Local buffer size in bytes
- `server_address` (str): Server address in format “hostname:port”

### Returns
- `int`: Status code (0 = success, non-zero = error code)

### Request Example
```python
# Initialize with dummy client
store.setup_dummy(1024*1024*256, 1024*1024*64, "localhost:8080")
```
```

--------------------------------

### MooncakeStore::setup

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/api-reference/rust/mooncake-store.md

Initializes the store client and establishes connections. This method requires several configuration parameters to set up the connection and buffer sizes.

```APIDOC
## MooncakeStore::setup

### Description
Initialise the store client and establish connections.

### Parameters
- `local_hostname` (string): IP/hostname for this node.
- `metadata_server` (string): metadata URI, for example: `"http://127.0.0.1:8080/metadata"` or `"etcd://127.0.0.1:2379"`.
- `global_segment_size` (usize): per-segment size in bytes.
- `local_buffer_size` (usize): local staging buffer size in bytes.
- `protocol` (string): transport protocol string (for example `"tcp"` / `"rdma"`).
- `device_name` (string): device selector; empty string means auto-select (when supported by the backend).
- `master_server_addr` (string): `mooncake_master` address, e.g. `"127.0.0.1:50051"`.

### Returns
- `Result<(), StoreError>`: `Ok(())` on success, otherwise `StoreError::OperationFailed(code)`.
```

--------------------------------

### Serve Documentation Locally (Default Port)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/README.md

Start a local HTTP server to view the generated documentation in your browser. The server will be accessible at http://localhost:8000.

```bash
python -m http.server -d build/html/
```

--------------------------------

### Start Mooncake Master with SSD Offload

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/performance/ssd-offload-benchmark-results.md

Starts the Mooncake master server with SSD offload enabled. This configuration is part of the setup for utilizing SSD for offloading data, requiring specific ports for HTTP metadata and metrics.

```bash
mooncake_master \
    -enable_offload=true \
    -http_metadata_server_port=8081 \
    -metrics_port=9004 \
    -logtostderr
```

--------------------------------

### Get Inner Transfer Engine Instance

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/transfer-engine.md

Retrieves the inner transfer engine instance. This instance can be reused, for example, with the mooncake store.

```python
get_engine()
```

--------------------------------

### P2P Hello World Example

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Demonstrates setting up and using the MooncakeDistributedStore with P2P handshake for decentralized peer discovery, avoiding the need for an HTTP metadata server.

```APIDOC
## P2P Hello World Example

### Description
This example illustrates how to initialize and configure `MooncakeDistributedStore` using the `P2PHANDSHAKE` option for the metadata server, enabling decentralized peer discovery without requiring a separate HTTP metadata server. It then proceeds to store and retrieve a simple key-value pair.

### Method
```python
from mooncake.store import MooncakeDistributedStore

store = MooncakeDistributedStore()
store.setup(
    "localhost",           # Your node's ip address
    "P2PHANDSHAKE",              # P2P handshake (no HTTP metadata)
    512*1024*1024,                # 512MB segment size
    128*1024*1024,                # 128MB local buffer
    "tcp",                       # Use TCP (RDMA for high performance)
    "",                          # Leave empty; Mooncake auto-picks RDMA devices when needed
    "localhost:50051"        # Master service
)

store.put("hello_key", b"Hello, Mooncake Store!")
print(store.get("hello_key").decode())
store.close()
```

### Parameters for `setup()`

- **node_address** (string) - Required - The IP address of your node.
- **metadata_server** (string) - Required - Set to `"P2PHANDSHAKE"` to enable peer-to-peer handshake for metadata.
- **segment_size** (int) - Required - The size of data segments in bytes.
- **local_buffer** (int) - Required - The size of the local buffer in bytes.
- **protocol** (string) - Required - The network protocol to use (e.g., "tcp", "rdma", "efa").
- **rdma_devices** (string) - Optional - Comma-separated list of RDMA devices. Leave empty for auto-discovery.
- **master_service** (string) - Required - The address of the master service (e.g., "host:port").

### Methods

- **`put(key: str, value: bytes)`**: Stores a key-value pair.
- **`get(key: str)`**: Retrieves the value for a given key.
- **`close()`**: Closes the store connection.
```

--------------------------------

### Serve Documentation Locally (Custom Port)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/README.md

Start a local HTTP server on a specified port (e.g., 3000) if the default port 8000 is in use. This allows you to view the documentation at http://localhost:3000.

```bash
python -m http.server 3000 -d build/html/
```

--------------------------------

### Get a PyTorch Tensor from Mooncake Store

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Retrieves a single PyTorch tensor by its key. Requires `torch` to be installed. Returns the tensor or `None` if not found.

```python
import torch
from mooncake.store import MooncakeDistributedStore

store = MooncakeDistributedStore()
store.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 128*1024*1024, "tcp", "", "localhost:50051")

# Store a tensor
tensor = torch.randn(100, 100)
store.put_tensor("my_tensor", tensor)

# Retrieve the tensor
retrieved_tensor = store.get_tensor("my_tensor")
if retrieved_tensor is not None:
    print(f"Retrieved tensor with shape: {retrieved_tensor.shape}")
```

--------------------------------

### Install Dependencies and Build Conductor

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/conductor/conductor-architecture-design.md

Run these commands in the conductor-ctrl directory to fetch dependencies and build the Conductor executable.

```bash
cd mooncake-conductor/conductor-ctrl
go mod tidy
go build -o mooncake_conductor .
```

--------------------------------

### Python RDMA Setup: Auto-select Devices

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Configures the Mooncake Distributed Store for RDMA protocol, automatically selecting available network devices. This example uses default discovery settings.

```python
from mooncake.store import MooncakeDistributedStore as S
s = S()
s.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 128*1024*1024, "rdma", "", "localhost:50051")
```

--------------------------------

### Initialize Mooncake Store with Dummy Client

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Sets up the store using a dummy client for testing purposes. Requires memory pool size, local buffer size, and server address.

```python
# Initialize with dummy client
store.setup_dummy(1024*1024*256, 1024*1024*64, "localhost:8080")
```

--------------------------------

### Start Transfer Engine Receiver (Server)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/quick-start.md

Initializes the Mooncake Transfer Engine server, registers a memory buffer, and sends buffer information to a client. This code is intended for the server-side setup.

```python
import numpy as np
import zmq
from mooncake.engine import TransferEngine

def main():
    # Initialize ZMQ context and socket
    context = zmq.Context()
    socket = context.socket(zmq.PUSH)
    socket.bind("tcp://*:5555")  # Bind to port 5555 for buffer info

    HOSTNAME = "localhost" # localhost for simple demo
    METADATA_SERVER = "P2PHANDSHAKE" # [ETCD_SERVER_URL, P2PHANDSHAKE, ...]
    PROTOCOL = "tcp" # use "rdma" on machines with RDMA devices configured
    DEVICE_NAME = "" # auto discovery if empty

    # Initialize server engine
    server_engine = TransferEngine()
    server_engine.initialize(
        HOSTNAME,
        METADATA_SERVER,
        PROTOCOL,
        DEVICE_NAME
    )
    session_id = f"{HOSTNAME}:{server_engine.get_rpc_port()}"

    # Allocate memory on server side (1MB buffer)
    server_buffer = np.zeros(1024 * 1024, dtype=np.uint8)
    server_ptr = server_buffer.ctypes.data
    server_len = server_buffer.nbytes

    # Register memory with Mooncake so the target address is advertised
    ret_value = server_engine.register_memory(server_ptr, server_len)
    if ret_value != 0:
        print("Mooncake memory registration failed.")
        raise RuntimeError("Mooncake memory registration failed.")

    print(f"Server initialized with session ID: {session_id}")
    print(f"Server buffer address: {server_ptr}, length: {server_len}")

    # Send buffer info to client
    buffer_info = {
        "session_id": session_id,
        "ptr": server_ptr,
        "len": server_len
    }
    socket.send_json(buffer_info)
    print("Buffer information sent to client")

    # Keep server running
    try:
        while True:
            input("Press Ctrl+C to exit...")
    except KeyboardInterrupt:
        print("\nShutting down server...")
    finally:
        # Cleanup
        ret_value = server_engine.unregister_memory(server_ptr)
        if ret_value != 0:
            print("Mooncake memory deregistration failed.")
            raise RuntimeError("Mooncake memory deregistration failed.")

        socket.close()
        context.term()

if __name__ == "__main__":
    main()

```

--------------------------------

### Start vLLM Decoder Node with Tensor Parallelism

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/vllm-integration/vllm-integration-v1.0.md

Launch a vLLM decoder node with tensor parallelism enabled, consuming KV caches via MooncakeConnector. This setup is for the decoding stage.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --port 8020 \
  --tensor-parallel-size 8 \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'
```

--------------------------------

### Smoke Test Generation Request

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/sglang-integration/hicache-quick-start.md

Sends a POST request to the router to test the generation capabilities of the disaggregated SGLang setup. This example uses a simple prompt and a temperature of 0 for deterministic output.

```bash
curl -X POST http://127.0.0.1:8000/generate \
    -H "Content-Type: application/json" \
    -d '{
  "text": "Let me tell you a long story ",
  "sampling_params": {
    "temperature": 0
  }
}'
```

--------------------------------

### Start Initiator Node

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/transfer-engine/index.md

Use this command to start the initiator node. Ensure `--metadata_server` points to your metadata storage and `--segment_id` matches the target node's `--local_server_name`. Optionally specify `--device_name` for RDMA or use `--auto-discovery`.

```bash
./transfer_engine_bench --metadata_server=etcd://10.0.0.1:2379 \
                           --segment_id=TARGET_NAME \
                           [--local_server_name=INITIATOR_NAME] \
                           [--device_name=erdma_1 | --auto-discovery]
```

--------------------------------

### Initialize and Perform Synchronous Data Transfer

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/transfer-engine.md

Demonstrates the basic setup of the TransferEngine, including initialization with connection details and transport protocol. It then shows how to register memory, prepare data, and perform a synchronous write operation to a remote node.

```python
import numpy as np
from mooncake.engine import TransferEngine
import os

# Create transfer engine instance
engine = TransferEngine()

# Initialize with basic configuration
engine.initialize(
    "127.0.0.1:12345", # local hostname
    "127.0.0.1:2379", # metadata server
    "rdma", # transport protocol
    "" # device name
)

# Allocate and initialize client buffer (1MB)
client_buffer = np.ones(1024 * 1024, dtype=np.uint8)  # Fill with ones
buffer_data = client_buffer.ctypes.data
buffer_data_len = client_buffer.nbytes

engine.register_memory(buffer_data, buffer_data_len)

# Get Remote Addr from ZMQ or upper-layer inference framework
remote_addr = REMOTE_ADDR

# Transfer data to remote node
ret = engine.transfer_sync_write(
    "127.0.0.1:12346", # target hostname
    data, # buffer
    remote_addr, # peer buffer address
    data_len # length
)

if ret == 0:
    print("Data transfer completed successfully")
else:
    print(f"Data transfer failed with code {ret}")

engine.unregister_memory(data)
```

--------------------------------

### Batch Zero-Copy Put and Get for Multiple Buffers Example

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Illustrates storing and retrieving multiple objects using multiple pre-registered buffers with batch_put_from_multi_buffers and batch_get_into_multi_buffers. Buffers must be registered and unregistered appropriately.

```python
tensor = torch.ones(10, 61, 128*1024, dtype=torch.int8)
data_ptr = tensor.data_ptr()
store.register_buffer(data_ptr, 10*61*128*1024)

target_tensor = torch.zeros(10, 61, 128*1024, dtype=torch.int8)
target_data_ptr = target_tensor.data_ptr()
store.register_buffer(target_data_ptr, 10*61*128*1024)

all_local_addrs = []
all_remote_addrs = []
all_sizes = []
keys = []
for block_i in range(10):
  local_addrs = []
  remote_addrs = []
  sizes = []
  for _ in range(61):
    local_addrs.append(data_ptr)
    remote_addrs.append(target_data_ptr)
    sizes.append(128*1024)
    data_ptr += 128*1024
    target_data_ptr += 128*1024
  all_local_addrs.append(local_addrs)
  all_remote_addrs.append(remote_addrs)
  all_sizes.append(sizes)
  keys.append(f"kv_{rank}_{block_i}")

config = ReplicateConfig()
config.prefer_alloc_in_same_node = True
store.batch_put_from_multi_buffers(keys, all_local_addrs, all_sizes, config)
store.batch_get_into_multi_buffers(keys, all_remote_addrs, all_sizes, True)

store.unregister_buffer(tensor.data_ptr())
store.unregister_buffer(target_tensor.data_ptr())
```

--------------------------------

### Navigate to Docs Directory

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/README.md

Change the current directory to the 'docs' folder to begin the build process.

```bash
cd docs
```

--------------------------------

### Batch Get PyTorch Tensors from Mooncake Store

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Retrieves multiple PyTorch tensors using a list of keys. Requires `torch` to be installed. The returned list contains tensors or `None` for missing keys.

```python
import torch
from mooncake.store import MooncakeDistributedStore

store = MooncakeDistributedStore()
store.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 128*1024*1024, "tcp", "", "localhost:50051")

# Store tensors
tensor1 = torch.randn(100, 100)
tensor2 = torch.randn(50, 50)
store.put_tensor("tensor1", tensor1)
store.put_tensor("tensor2", tensor2)

# Retrieve multiple tensors
tensors = store.batch_get_tensor(["tensor1", "tensor2", "nonexistent"])
for i, tensor in enumerate(tensors):
    if tensor is not None:
        print(f"Tensor {i} shape: {tensor.shape}")
    else:
        print(f"Tensor {i} not found")
```

--------------------------------

### Complete Zero-Copy Workflow Example

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/python-api-reference/mooncake-store.md

Demonstrates the full zero-copy workflow, including buffer registration, storing data with `put_from`, retrieving data with `get_into`, and unregistering buffers. Ensure buffers are registered before zero-copy operations to prevent memory corruption.

```python
import numpy as np
from mooncake.store import MooncakeDistributedStore

# Initialize store with RDMA protocol for maximum performance
store = MooncakeDistributedStore()
store.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "tcp", "", "localhost:50051")

# Create data to store
original_data = np.random.randn(1000, 1000).astype(np.float32)
buffer_ptr = original_data.ctypes.data
size = original_data.nbytes

# Step 1: Register the buffer
result = store.register_buffer(buffer_ptr, size)
if result != 0:
    raise RuntimeError(f"Failed to register buffer: {result}")

# Step 2: Zero-copy store
result = store.put_from("large_tensor", buffer_ptr, size)
if result == 0:
    print(f"Successfully stored {size} bytes with zero-copy")
else:
    raise RuntimeError(f"Store failed with code: {result}")

# Step 3: Pre-allocate buffer for retrieval
retrieved_data = np.empty((1000, 1000), dtype=np.float32)
recv_buffer_ptr = retrieved_data.ctypes.data
recv_size = retrieved_data.nbytes

# Step 4: Register receive buffer
result = store.register_buffer(recv_buffer_ptr, recv_size)
if result != 0:
    raise RuntimeError(f"Failed to register receive buffer: {result}")

# Step 5: Zero-copy retrieval
bytes_read = store.get_into("large_tensor", recv_buffer_ptr, recv_size)
if bytes_read > 0:
    print(f"Successfully retrieved {bytes_read} bytes with zero-copy")
    # Verify the data
    print(f"Data matches: {np.array_equal(original_data, retrieved_data)}")
else:
    raise RuntimeError(f"Retrieval failed with code: {bytes_read}")

# Step 6: Clean up - unregister both buffers
store.unregister_buffer(buffer_ptr)
store.unregister_buffer(recv_buffer_ptr)
store.close()
```

--------------------------------

### Create P2PStore Instance

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/p2p-store.md

Initializes a P2PStore, starting an internal Transfer Engine service. Requires metadata server URI and a unique local segment name.

```go
func NewP2PStore(metadataUri string, localSegmentName string) (*P2PStore, error)
```

--------------------------------

### Install SGLang (Default CUDA)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/sglang-integration-v1.md

Install SGLang using pip and uv for faster installation. This command installs the default CUDA version.

```bash
pip install --upgrade pip
pip install uv
uv pip install sglang
```

--------------------------------

### Start Simulated Training Node for P2P Store

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/p2p-store.md

Initiates a simulated training node that registers a local file and makes it public within the cluster. Ensure MC_GID_INDEX is set and the metadata server is accessible.

```bash
# This is 10.0.0.2
export MC_GID_INDEX=n    # NOTE that n is integer
./p2p-store-example --cmd=trainer \
                       --metadata_server=10.0.0.1:2379 \
                       --local_server_name=10.0.0.2:12345 \
   
```

--------------------------------

### Install Mooncake Transfer Engine

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/vllm-integration/disagg-prefill-decode.md

Installs the Mooncake transfer engine package. Ensure prerequisites are met before installation.

```bash
pip3 install mooncake-transfer-engine
```

--------------------------------

### Install yalantinglibs

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/transfer-engine/kunpeng_ub_transport.md

Builds and installs the yalantinglibs from source, which is a required dependency. It clones the repository, configures with CMake, and installs to /usr/local.

```bash
# Install yalantinglibs (required)
cd /tmp
git clone https://github.com/alibaba/yalantinglibs.git
cd yalantinglibs
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local
make -j$(nproc)
sudo make install
```

--------------------------------

### Install SSH Dependencies

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/deployment/nvmf-ssd-deployment-guide.md

Installs the necessary Python package for SSH operations. Ensure you have Python 3 and pip installed.

```bash
python3 -m pip install "paramiko>=3.4.0"
```

--------------------------------

### Start Target Node for Scenario Testing

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/transfer-engine/ascend_transport.md

Use this command to start the target node for scenario testing. Ensure the local server name and device ID are correctly configured.

```bash
./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target --block_size=8388608 --batch_size=32
```

--------------------------------

### Start Mooncake Store Service (Real Client)

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/getting_started/examples/sglang-integration/hicache-integration-v1.md

Starts the Mooncake store service, which acts as the real client for the dummy client mode. It allocates a global segment size and listens on a default port for internal RPC.

```bash
mooncake_client --global_segment_size=4GB
```

--------------------------------

### Install Mooncake Common Library

Source: https://github.com/kvcache-ai/mooncake/blob/main/mooncake-common/src/CMakeLists.txt

Installs the mooncake_common library to the 'lib' directory if BUILD_SHARED_LIBS is enabled. This makes the library available for use by other projects after installation.

```cmake
if (BUILD_SHARED_LIBS)
  install(TARGETS mooncake_common DESTINATION lib)
endif()
```

--------------------------------

### Configure and Run Conductor

Source: https://github.com/kvcache-ai/mooncake/blob/main/docs/source/design/conductor/conductor-architecture-design.md

Set environment variables for configuration path and log level, then execute the Conductor binary.

```bash
export CONDUCTOR_CONFIG_PATH=../example/conductor_config.json
export CONDUCTOR_LOG_LEVEL=INFO
./mooncake_conductor
```

--------------------------------

### Install Etcd Wrapper Shared Library

Source: https://github.com/kvcache-ai/mooncake/blob/main/mooncake-common/etcd/CMakeLists.txt

Configures CMake to install the built Etcd wrapper shared library into the 'lib' directory of the installation prefix.

```cmake
install(
    FILES ${ETCD_WRAPPER_LIB}
    DESTINATION lib
)
```

--------------------------------

### Set Install RPATH

Source: https://github.com/kvcache-ai/mooncake/blob/main/mooncake-transfer-engine/src/CMakeLists.txt

Enables the use of the link path for the installation RPATH.

```cmake
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
```