### Vast.ai CLI Quick Start Guide Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md A step-by-step guide to authenticate, set up SSH keys, search for offers, create an instance, verify its status, and manage files and instances. ```bash vastai set api-key # Authenticate (one-time); Create API Key in account at https://console.vast.ai/manage-keys/ vastai show user # Verify auth + check balance vastai create ssh-key ~/.ssh/id_ed25519.pub # Register SSH key (do BEFORE create) vastai search offers 'gpu_name=RTX_4090 num_gpus=1 verified=true direct_port_count>=1 rentable=true' -o 'dlperf_usd-' # Note the offer ID from the output vastai create instance --image pytorch/pytorch:@vastai-automatic-tag --disk 20 --ssh --direct # Automatically grab image tag; Response: {"success": true, "new_contract": } vastai show instance # Poll until actual_status == "running" (see Instance status values below) vastai ssh-url # Get SSH connection string vastai copy local:./data/ :/workspace/ # Upload files vastai destroy instance -y # Clean up (stops all billing; -y skips confirmation) ``` -------------------------------- ### Startup Command Configuration Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CONFIGURATION.md Specify a command to run when the instance starts. ```python config = InstanceConfig( onstart="bash /setup.sh && python train.py", ) ``` -------------------------------- ### Complete Workflow Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Demonstrates a full workflow from searching for offers to creating, monitoring, and destroying an instance. ```bash # 1. Search for offers vastai search offers 'gpu_name=RTX_4090 num_gpus>=2' --limit 10 # 2. Create instance from top offer (ID 12345) vastai create instance 12345 \ --image pytorch/pytorch:latest \ --disk 50 \ --label "ml-training" \ --jupyter-lab # 3. Get SSH URL vastai show-ssh-url 54321 # 4. Monitor instance vastai show instance 54321 vastai logs 54321 --tail 20 # 5. Execute command vastai execute 54321 "python train.py" # 6. Stop when done vastai stop instance 54321 # 7. Resume later vastai start instance 54321 # 8. Clean up vastai destroy instance 54321 ``` -------------------------------- ### Python AsyncClient Search Offers Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Example of asynchronously searching for offers, specifically for RTX_4090 GPUs, and printing their details. Requires asyncio and Query import. ```python import asyncio from vastai import AsyncClient from vastai.data.query import Query async def main(): client = AsyncClient() query = Query.where_gpu_name("RTX_4090") offers = await client.search(query) for offer in offers: print(f"GPU: {offer.gpu_name}, Price: ${offer.dph_total}/hr") asyncio.run(main()) ``` -------------------------------- ### Python AsyncClient Create Instance Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Example of asynchronously creating a new instance with a specified image and disk size, then printing its ID. Requires asyncio. ```python async def main(): client = AsyncClient() config = InstanceConfig(image="pytorch/pytorch", disk=32) instance = await client.create_instance(12345, config) print(f"Created: {instance.id}") asyncio.run(main()) ``` -------------------------------- ### Install Vast.ai Python SDK Source: https://github.com/vast-ai/vast-cli/blob/master/vastai_sdk/SKILL.md Install the Vast.ai Python SDK using pip. For serverless and async support, install with the 'serverless' extra. ```bash pip install vastai ``` ```bash pip install "vastai[serverless]" ``` -------------------------------- ### Install Vast.ai Python SDK Source: https://github.com/vast-ai/vast-cli/blob/master/sdk-wrapper/README.md Install the Vast.ai Python SDK using pip. This command installs the SDK and CLI in a single package. ```bash pip install vastai-sdk ``` -------------------------------- ### Common Patterns: Launching a Job Source: https://github.com/vast-ai/vast-cli/blob/master/vastai_sdk/SKILL.md Example demonstrating how to find the cheapest available GPUs and launch a job. Use `help()` to explore method signatures. ```python # Find cheapest 4x RTX 4090 and launch a job from vastai import VastAI vast = VastAI() offers = vast.search_offers(query='gpu_name=RTX_4090 num_gpus=4 reliability>0.99') cheapest = min(offers, key=lambda o: o['dph_total']) result = vast.create_instance(id=cheapest['id'], image="pytorch/pytorch:latest", disk=100) print(f"Launched instance: {result['new_contract']}") # Use help() to explore method signatures help(vast.search_offers) help(vast.create_instance) ``` -------------------------------- ### Build and Run pip Install Tests Source: https://github.com/vast-ai/vast-cli/blob/master/tests/pip_install/README.md Builds the necessary wheels for vastai and vastai-sdk, then executes the integration tests for pip installation scenarios. Assumes a clean environment. ```bash pip install build poetry-core python3 -m build --wheel cd sdk-wrapper && python3 -m build --wheel && cd .. bash tests/pip_install/test_install_scenarios.sh ``` -------------------------------- ### Vast.ai CLI Help Commands Source: https://github.com/vast-ai/vast-cli/blob/master/README.md Examples of how to access help documentation for the Vast.ai CLI and its subcommands. ```bash vastai search offers --help ``` ```bash vastai create instance --help ``` -------------------------------- ### Python SyncInstance Start Method Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Starts a stopped synchronous instance. This method returns None. ```python def start() -> None ``` -------------------------------- ### Python AsyncClient Constructor Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Demonstrates creating an AsyncClient instance, showing both default initialization and custom connection limit. ```python from vastai import AsyncClient # Create client client = AsyncClient() # With custom connection limit client = AsyncClient(connection_limit=50) ``` -------------------------------- ### Install Vast.ai Package Source: https://github.com/vast-ai/vast-cli/blob/master/README.md Install the Vast.ai Python package using pip. The `vastai-sdk` name is also supported for backward compatibility. ```bash pip install vastai ``` -------------------------------- ### Synchronous Client Workflow Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Demonstrates a typical synchronous workflow using SyncClient to search for offers, create an instance, and manage it. ```python from vastai import SyncClient from vastai.data.query import Query, InstanceConfig # Create client client = SyncClient() # Search offers query = Query.where(gpu_name="RTX_4090", num_gpus=1) offers = client.search(query, limit=5) if offers: # Use first offer offer = offers[0] print(f"Selected: {offer.gpu_name} @ ${offer.dph_total}/hr") # Create instance config = InstanceConfig( image="pytorch/pytorch:latest", disk=32, ) instance = offer.create_instance(config) print(f"Created instance: {instance.id}") print(f"SSH: {instance.ssh_url}") # List all instances all_instances = client.show_instances() print(f"Total instances: {len(all_instances)}") # Stop instance instance.stop() # Destroy instance instance.destroy() ``` -------------------------------- ### Install Vast.ai CLI Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Install the Vast.ai CLI tool using pip. You can use the primary package name or an alternate SDK name. ```bash pip install vastai ``` ```bash pip install vastai-sdk ``` -------------------------------- ### BenchmarkConfig Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Creates a BenchmarkConfig instance with a fixed dataset, specifying the number of runs and concurrency level. ```python from vastai import BenchmarkConfig benchmark = BenchmarkConfig( dataset=[ {"prompt": "test1"}, {"prompt": "test2"}, {"prompt": "test3"}, ], runs=10, concurrency=4, do_warmup=True, ) ``` -------------------------------- ### Vast.ai CLI Usage Examples Source: https://github.com/vast-ai/vast-cli/blob/master/README.md Demonstrates various commands for managing Vast.ai resources via the CLI, including searching offers, creating instances, and managing their lifecycle. ```bash vastai search offers 'gpu_name=RTX_4090 num_gpus>=4' ``` ```bash vastai create instance 12345 --image pytorch/pytorch --disk 32 --ssh --direct ``` ```bash vastai show instances ``` ```bash vastai stop instance 12345 ``` ```bash vastai destroy instance 12345 ``` -------------------------------- ### Vast.ai CLI Search Query Syntax Examples Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Examples of using filter expressions in Vast.ai CLI search commands, including exact matches, numeric comparisons, and region/price filtering. ```bash # Examples 'gpu_name=RTX_4090 num_gpus=1' # Exact match + numeric 'gpu_ram>=48 reliability>0.95' # Greater-than filters 'geolocation=EU dph_total<=2.0' # Region + price cap ``` -------------------------------- ### Run pip Install Tests with Custom Wheel Paths Source: https://github.com/vast-ai/vast-cli/blob/master/tests/pip_install/README.md Executes the pip installation integration tests using custom wheel file paths for vastai and vastai-sdk. This allows testing with pre-built or specific wheel versions. ```bash VASTAI_WHL=/path/to/vastai.whl SDK_WHL=/path/to/vastai_sdk.whl \ bash tests/pip_install/test_install_scenarios.sh ``` -------------------------------- ### WorkerConfig Constructor Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Configure a serverless worker with model server details, handlers, logging, and benchmarking options. ```python from vastai import WorkerConfig, HandlerConfig, LogActionConfig, BenchmarkConfig config = WorkerConfig( model_server_url="http://localhost:8000", model_server_port=8000, model_log_file="/var/log/model.log", model_healthcheck_url="/health", benchmark_route="/benchmark", max_sessions=20, handlers=[ HandlerConfig( route="/v1/completions", healthcheck="/health", allow_parallel_requests=True, ), HandlerConfig( route="/v1/embeddings", healthcheck="/health", ), ], log_action_config=LogActionConfig( on_load=["Model loaded successfully"], on_error=["Error during processing"], ), benchmark_data=[ {"input": "test1", "expected": "output1"}, {"input": "test2", "expected": "output2"}, ], ) ``` -------------------------------- ### LogActionConfig Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Initializes LogActionConfig with lists of messages for model loading, error handling, and informational logging. ```python from vastai import LogActionConfig log_config = LogActionConfig( on_load=[ "Model initialized", "Loading weights...", ], on_error=[ "Request processing failed", "Invalid input detected", ], on_info=[ "Processing request", "Generating response", ], ) ``` -------------------------------- ### Control instance lifecycle Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Commands to manage the state of an instance, such as starting, stopping, rebooting, or destroying it. ```bash vastai start instance # Start stopped instance ``` ```bash vastai stop instance # Stop (preserves disk, no GPU charges) ``` ```bash vastai reboot instance # Stop + start ``` ```bash vastai destroy instance -y # Delete permanently (irreversible; -y required for non-interactive use) ``` ```bash vastai destroy instances -y # Batch delete (-y skips confirmation prompt) ``` -------------------------------- ### Vast.ai SDK Usage Examples Source: https://github.com/vast-ai/vast-cli/blob/master/README.md Demonstrates how to use the Vast.ai Python SDK to interact with Vast.ai services, including searching offers and managing instances. ```python from vastai import VastAI vast = VastAI() # uses VAST_API_KEY env var, or pass api_key="..." vast.search_offers(query='gpu_name=RTX_4090 num_gpus>=4') vast.show_instances() vast.start_instance(id=12345) vast.stop_instance(id=12345) ``` -------------------------------- ### AsyncInstance.start Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Asynchronously starts a stopped instance. This action resumes the execution of the instance, making it available for use. ```APIDOC ## start() ### Description Start the instance. ### Returns None ``` -------------------------------- ### Use recommended Vast.ai images Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Examples of using recommended Vast.ai images, including a base image and specialized images for PyTorch and Linux desktop. ```bash vastai/base-image:@vastai-automatic-tag # Minimal Ubuntu base ``` ```bash vastai/pytorch:@vastai-automatic-tag # PyTorch + CUDA ``` ```bash vastai/linux-desktop:@vastai-automatic-tag # Linux desktop (VNC/RDP) ``` -------------------------------- ### Search Offers with Filters Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Provides examples of searching for offers using various filters like budget, performance, and provider verification. ```bash # Budget search (under $0.25/hr) vastai search offers 'dph_total<=0.25' --limit 20 # High-performance search vastai search offers 'num_gpus>=4 gpu_ram>=48000' --limit 5 # Specific provider search vastai search offers 'verified=true' --limit 50 # All offer types vastai search offers --type bid vastai search offers --type reserved vastai search offers --type on-demand ``` -------------------------------- ### Test GPU Offer Search Source: https://github.com/vast-ai/vast-cli/blob/master/README.md Perform a basic search for available GPU offers using the CLI to verify your setup. ```bash vastai search offers --limit 3 ``` -------------------------------- ### Asynchronous Client Workflow Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Illustrates an asynchronous workflow using AsyncClient, including searching offers, creating instances, and managing them within an asyncio event loop. ```python import asyncio from vastai import AsyncClient from vastai.data.query import Query, InstanceConfig async def main(): # Create client client = AsyncClient() # Search offers query = Query.where(gpu_name="RTX_4090") offers = await client.search(query, limit=5) if offers: offer = offers[0] print(f"Selected: {offer.gpu_name}") # Create instance config = InstanceConfig(image="pytorch/pytorch", disk=32) instance = await offer.create_instance(config) print(f"Created: {instance.id}") # Get all instances all_instances = await client.show_instances() print(f"Total: {len(all_instances)}") # Stop and destroy await instance.stop() await instance.destroy() await client.close() asyncio.run(main()) ``` -------------------------------- ### HandlerConfig Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Instantiates HandlerConfig with specific route, healthcheck, parallel request allowance, queue time, and benchmark configuration. ```python from vastai import HandlerConfig, BenchmarkConfig handler = HandlerConfig( route="/v1/completions", healthcheck="/health", allow_parallel_requests=True, max_queue_time=60.0, benchmark_config=BenchmarkConfig( dataset=[ {"prompt": "Hello", "expected": "Hi"}, {"prompt": "Goodbye", "expected": "Bye"}, ], runs=5, concurrency=2, do_warmup=True, ), ) ``` -------------------------------- ### Async Client Context Manager Usage Example Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Shows how to use the AsyncClient with an 'async with' statement for cleaner management of client lifecycle, automatically handling closing. ```python import asyncio from vastai import AsyncClient from vastai.data.query import Query async def main(): async with AsyncClient() as client: query = Query.where_gpu_name("RTX_4090") offers = await client.search(query) print(f"Found {len(offers)} offers") # Client automatically closes on exit asyncio.run(main()) ``` -------------------------------- ### Configure Instance Image Selection Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CONFIGURATION.md Shows how to specify the Docker image for an instance using the `InstanceConfig` class, providing examples for official PyTorch and TensorFlow images. ```python from vastai.data.instance import InstanceConfig # Official PyTorch config = InstanceConfig(image="pytorch/pytorch:latest") # Official TensorFlow config = InstanceConfig(image="tensorflow/tensorflow:latest-gpu") ``` -------------------------------- ### Worker.run() Method Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Starts the worker server and begins handling requests. This asynchronous method should be awaited to activate the worker. ```APIDOC ### Methods #### async run() ```python async def run() -> None ``` Start the worker server and begin handling requests. **Example:** ```python import asyncio from vastai import Worker, WorkerConfig async def main(): config = WorkerConfig( model_server_url="http://localhost:8000", ) worker = Worker(config) await worker.run() asyncio.run(main()) ``` ``` -------------------------------- ### Search and Create Instance (Python SDK) Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/INDEX.md Demonstrates how to search for available GPU offers and then create a new instance using the high-level VastAI client. Requires an active internet connection and valid API key. ```python from vastai import VastAI from vastai.data.instance import InstanceConfig vast = VastAI() # Search offers = vast.search_offers(query="gpu_name=RTX_4090 num_gpus>=2") # Create if offers: config = InstanceConfig(image="pytorch/pytorch", disk=32) instance = vast.create_instance(offers[0]["id"], **config.__dict__) ``` -------------------------------- ### AsyncInstance Start Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Asynchronously starts a stopped instance. ```python async def start() -> None ``` -------------------------------- ### Simple Deployment Configuration Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Define a basic Vast.ai deployment with a name, source path, and version. This is the starting point for pushing your inference service to Vast.ai. ```python from vastai import Deployment deployment = Deployment( name="inference-service", path="./src", version="1.0.0", ) # Push to Vast.ai # deployment.push() ``` -------------------------------- ### Start a Stopped Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Start a stopped instance using its ID. ```python start_response = vast.start_instance(id=12345) ``` -------------------------------- ### launch_instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Launches an instance from search offers matching the given criteria. Requires GPU name, number of GPUs, and a Docker image URL. ```APIDOC ## launch_instance ### Description Launch the top instance from search offers matching the given criteria. ### Method Not specified (assumed to be a client-side SDK method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body * **gpu_name** (str) - Required - GPU name (e.g., "RTX_4090") * **num_gpus** (str) - Required - Number of GPUs * **image** (str) - Required - Docker image URL * **kwargs** (dict) - Optional - Additional parameters ### Request Example ```python launch_instance(gpu_name="RTX_4090", num_gpus="1", image="") ``` ### Response #### Success Response - Response dict with created instance details #### Response Example ```json { "instance_id": "", "status": "running" } ``` ``` -------------------------------- ### VastClient GET Request Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Executes a GET request to a specified subpath with optional query arguments, JSON data, and timeout. ```python get( subpath: str, query_args: Optional[Dict] = None, json_data: Optional[Dict] = None, timeout: Optional[float] = None ) -> requests.Response ``` -------------------------------- ### Initialize VastAI with API Key Methods Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CONFIGURATION.md Demonstrates three ways to initialize the VastAI client: using an environment variable, passing the API key directly to the constructor, or relying on a configuration file. ```python import os # Method 1: Environment variable os.environ["VAST_API_KEY"] = "vastai_key_xyz123" vast = VastAI() # Method 2: Explicit parameter vast = VastAI(api_key="vastai_key_xyz123") # Method 3: Config file (~/.config/vastai/vast_api_key) # Place API key in file, one key per line vast = VastAI() ``` -------------------------------- ### Search GPU Offers Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Search for available GPU offers with various filtering and sorting options. Examples demonstrate searching by GPU name, sorting by price, and using complex queries. ```bash vastai search offers [OPTIONS] [QUERY] ``` ```bash # Show top 5 RTX_4090 offers vastai search offers 'gpu_name=RTX_4090' --limit 5 ``` ```bash # Show offers sorted by price vastai search offers --order dph_total+ ``` ```bash # Complex query vastai search offers 'gpu_name=RTX_4090 num_gpus>=2 dph_total<=0.50' ``` ```bash # Raw JSON output vastai search offers --raw ``` ```bash # No default filters vastai search offers --no-default ``` -------------------------------- ### Async Search and Create Instance (Python SDK) Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/INDEX.md Shows how to asynchronously search for GPU offers and create an instance using the AsyncClient. This pattern is suitable for I/O-bound operations where concurrency is beneficial. Ensure the async context is properly managed. ```python import asyncio from vastai import AsyncClient from vastai.data.query import Query from vastai.data.instance import InstanceConfig async def main(): async with AsyncClient() as client: query = Query.where_gpu_name("RTX_4090") offers = await client.search(query) if offers: config = InstanceConfig(image="pytorch/pytorch") instance = await offers[0].create_instance(config) ``` -------------------------------- ### Manage Instances with VastAI Client Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/INDEX.md The VastAI client provides methods to list, retrieve, start, stop, reboot, and destroy instances. You need to instantiate the VastAI client to use these methods. ```python from vastai import VastAI vast = VastAI() # List all instances = vast.show_instances() # Get one instance = vast.show_instance(id=12345) # Control vast.stop_instance(id=12345) vast.start_instance(id=12345) vast.reboot_instance(id=12345) # Destroy vast.destroy_instance(id=12345) ``` -------------------------------- ### Get Async Client Session Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Gets or creates an aiohttp client session for asynchronous requests. Optionally uses a context manager. ```python async def _get_session(use_context: bool = False) -> aiohttp.ClientSession ``` -------------------------------- ### Take Instance Snapshot Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Create a snapshot of an instance and push it to the registry. This command requires the instance ID and may have additional options. ```bash vastai take-snapshot ID [OPTIONS] ``` -------------------------------- ### Launch a new instance Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Launches a new instance with specified GPU and image. The response includes the instance ID. ```bash vastai launch instance --gpu-name RTX_4090 --num-gpus 1 --image pytorch/pytorch ``` -------------------------------- ### Search and Create Instance (Asynchronous) Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/INDEX.md Illustrates how to perform asynchronous searches for offers and create instances using the `AsyncClient`. ```APIDOC ## Search and Create Instance (Asynchronous) ### Description This pattern demonstrates the asynchronous approach to searching for machine offers and subsequently creating an instance. It utilizes the `AsyncClient` for non-blocking operations. ### Method ```python import asyncio from vastai import AsyncClient from vastai.data.query import Query from vastai.data.instance import InstanceConfig async def main(): async with AsyncClient() as client: # Construct a query for offers query = Query.where_gpu_name("RTX_4090") offers = await client.search(query) # If offers are found, create an instance from the first one if offers: config = InstanceConfig(image="pytorch/pytorch") instance = await offers[0].create_instance(config) ``` **Note:** See **[SYNC_ASYNC_CLIENTS.md](SYNC_ASYNC_CLIENTS.md)** for more on `AsyncClient` patterns. ``` -------------------------------- ### get(subpath, query_args, json_data, timeout) Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Executes an HTTP GET request to the specified subpath with optional query arguments, JSON data, and timeout. ```APIDOC ## get(subpath, query_args, json_data, timeout) ### Description Executes an HTTP GET request to the specified subpath with optional query arguments, JSON data, and timeout. ### Method GET ### Endpoint /{subpath} ### Parameters #### Path Parameters - **subpath** (str) - Required - The subpath for the GET request. #### Optional Parameters - **query_args** (Optional[Dict]) - None - Dictionary of query arguments. - **json_data** (Optional[Dict]) - None - Dictionary representing JSON data to be sent (typically not used with GET, but included for completeness). - **timeout** (Optional[float]) - None - Request timeout in seconds. ### Returns requests.Response object representing the API response. ``` -------------------------------- ### Create Minimal Instance Configuration Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/TYPES.md Instantiate an InstanceConfig with only the required 'image' and 'disk' parameters. ```python from vastai.data.instance import InstanceConfig # Minimal config config = InstanceConfig( image="pytorch/pytorch:latest", disk=20, ) ``` -------------------------------- ### Vast.ai SDK and CLI Documentation Overview Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/README.md This document provides an overview of the available documentation for the Vast.ai SDK and CLI, guiding users to the most relevant sections for their needs. ```APIDOC ## Documentation Structure This documentation is organized into several files, each covering a specific aspect of the Vast.ai SDK and CLI: - **INDEX.md**: General overview, navigation, module organization, entry points, common usage patterns, and quick reference tables. - **API_REFERENCE.md**: Detailed reference for the high-level SDK (VastAI class) and low-level HTTP client (VastClient), covering instance management, search, offers, machine management, team management, SSH/API key management, endpoints, worker groups, and billing. - **SERVERLESS_CLIENT.md**: API documentation for the serverless client, focusing on endpoint inference, async operations, and managed endpoints. - **SYNC_ASYNC_CLIENTS.md**: Documentation for structured synchronous and asynchronous clients, including wrapper classes for offers and instances, and examples for batch operations. - **TYPES.md**: Definitions for all data classes and types used within the SDK, such as Query, InstanceConfig, Offer, and Instance. - **CONFIGURATION.md**: Information on configuration options, environment setup, API key resolution, constructor parameters, environment variables, and logging. - **ERRORS.md**: Reference for error handling, including standard exceptions, HTTP status codes, API response errors, and debugging techniques. - **DEPLOYMENT_SERVERSIDE.md**: Documentation for server-side workers and deployments, covering Deployment, Worker, and configuration classes. - **CLI_COMMANDS.md**: Reference for the command-line interface, including installation, configuration, and commands for search, instance management, SSH, keys, teams, endpoints, and billing. ## Key Features - **Complete Coverage**: Documents all exported classes, functions, parameters, signatures, environment variables, and configuration options. - **Developer-Friendly**: Provides markdown format, code examples, parameter tables, error solutions, and cross-references. - **Well-Organized**: Features logical module grouping, quick reference tables, common usage patterns, a navigation index, and a CLI command reference. - **Production-Ready**: Includes patterns for error handling, configuration best practices, debugging techniques, performance optimization, and security considerations. ## Quick Start Guide - **SDK Users**: Start with **INDEX.md**, then **API_REFERENCE.md**, **TYPES.md**, **CONFIGURATION.md**, and **ERRORS.md**. ``` -------------------------------- ### Create a New Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Create a new instance by accepting an offer and providing an InstanceConfig. The instance ID is printed upon successful creation. ```python from vastai import SyncClient, InstanceConfig client = SyncClient() config = InstanceConfig( image="pytorch/pytorch:latest", disk=32, label="my-gpu", ) instance = client.create_instance(offer_id=12345, config=config) print(f"Instance created: {instance.id}") ``` -------------------------------- ### start_instance(id) Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Initiates the startup process for a stopped instance, identified by its ID. ```APIDOC ## start_instance(id) ### Description Initiates the startup process for a stopped instance, identified by its ID. ### Method ```python start_instance(id: int) ``` ### Endpoint None ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **id** (int) - Required - Instance ID ### Request Example ```python response = vast.start_instance(12345) print(response) ``` ### Response #### Success Response (200) - **response** (dict) - Response dict confirming start action #### Response Example ```json { "status": "starting", "instance_id": "i-abcdef123456" } ``` ``` -------------------------------- ### Create Full Instance Configuration Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/TYPES.md Instantiate an InstanceConfig with all available parameters, including image, disk, labels, price, environment variables, and Jupyter Lab settings. ```python # Full config config = InstanceConfig( image="pytorch/pytorch:latest", disk=50, label="ml-training", price=0.50, env={"CUDA_VISIBLE_DEVICES": "0"}, onstart="python train.py", use_jupyter_lab=True, jupyter_dir="/workspace", python_utf8=True, ) ``` -------------------------------- ### Serverless Client for Inference Endpoints Source: https://github.com/vast-ai/vast-cli/blob/master/vastai_sdk/SKILL.md Utilize the Serverless client for inference endpoints. It automatically reads the API key from ~/.vast_api_key. Ensure you have installed the serverless extra: pip install "vastai[serverless]". ```python import asyncio from vastai import Serverless async def main(): serverless = Serverless() # reads ~/.vast_api_key # Get an endpoint endpoint = await serverless.get_endpoint("my-endpoint") # Make a request response = await serverless.request("/v1/completions", { "model": "Qwen/Qwen3-8B", "prompt": "Who are you?", "max_tokens": 100, "temperature": 0.7, }) text = response["response"]["choices"][0]["text"] print(text) asyncio.run(main()) ``` -------------------------------- ### Initialize VastAI Client with Explanation Mode Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CONFIGURATION.md Initialize the VastAI client in Python, enabling explanation mode based on an environment variable. ```python import os from vastai import VastAI vast = VastAI(explain=os.getenv("VAST_EXPLAIN") == "1") ``` -------------------------------- ### create_instance(id, image, disk, **kwargs) Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Creates a new instance from a specified contract offer ID, with options for Docker image, disk size, price, labels, environment variables, and startup commands. ```APIDOC ## create_instance(id, image, disk, **kwargs) ### Description Creates a new instance from a specified contract offer ID, with options for Docker image, disk size, price, labels, environment variables, and startup commands. ### Method ```python create_instance( id: int, image: Optional[str] = None, disk: float = 10, **kwargs ) ``` ### Endpoint None ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **id** (int) - Required - Offer ID - **image** (Optional[str]) - None - Docker image URL - **disk** (float) - 10 - Disk allocation in GiB - **price** (Optional[float]) - None - Max price to accept - **label** (Optional[str]) - None - Instance label - **env** (Optional[dict]) - None - Environment variables - **onstart_cmd** (Optional[str]) - None - Command to run on startup - **jupyter_lab** (bool) - False - Enable Jupyter Lab - **force** (bool) - False - Force creation even if unavailable ### Request Example ```python response = vast.create_instance( id=12345, image="pytorch/pytorch", disk=32, label="my-instance", jupyter_lab=True, ) ``` ### Response #### Success Response (200) - **response** (dict) - Response dict with instance details #### Response Example ```json { "instance_id": "i-abcdef123456", "status": "created", ... } ``` ``` -------------------------------- ### Create instance with vLLM and environment variables Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Creates an instance using the vLLM image, specifying disk size, SSH access, direct connection, and setting model name and Hugging Face token via environment variables. ```bash vastai create instance --image vastai/vllm:@vastai-automatic-tag --disk 40 --ssh --direct \ --env '-e MODEL_NAME=Qwen/Qwen2.5-3B-Instruct -e HF_TOKEN=hf_xxx' ``` -------------------------------- ### Start a Stopped Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Resume a previously stopped compute instance by providing its ID. ```bash vastai start instance ID ``` -------------------------------- ### Get Configuration Value Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Retrieves a specific configuration value from the Vast.ai CLI settings. ```bash vastai get api-key ``` -------------------------------- ### Initialize VastAI with Constructor Options Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CONFIGURATION.md Shows various ways to initialize the VastAI client using different combinations of constructor parameters for standard, debug, custom server, and raw output modes. ```python from vastai import VastAI # Standard initialization vast = VastAI() # Debug mode vast = VastAI(explain=True, quiet=False) # Custom server vast = VastAI(server_url="https://api-beta.vast.ai") # Raw JSON output vast = VastAI(raw=True) # All options vast = VastAI( api_key="key_123", server_url="https://api.vast.ai", retry=5, raw=True, explain=True, quiet=False, curl=True, ) ``` -------------------------------- ### Create a New Instance Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Create a new GPU instance specifying the offer ID, desired Docker image, disk size, and SSH access. ```bash vastai create instance --image pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime --disk 20 --ssh --direct ``` -------------------------------- ### Run Serverless Worker Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SERVERLESS_CLIENT.md Start the serverless worker process. This method should be called after the worker has been initialized with its configuration. ```python async def run() ``` -------------------------------- ### Create New Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Create a new compute instance from an offer. Customize image, disk size, labels, and enable features like Jupyter Lab or SSH. ```bash vastai create instance OFFER_ID [OPTIONS] ``` ```bash # Create with PyTorch image vastai create instance 12345 --image pytorch/pytorch --disk 32 ``` ```bash # With Jupyter Lab vastai create instance 12345 --image pytorch/pytorch --jupyter-lab --disk 50 ``` ```bash # Set environment variables vastai create instance 12345 --env CUDA_VISIBLE_DEVICES=0 --env BATCH_SIZE=32 ``` ```bash # Max price bid vastai create instance 12345 --price 0.50 ``` -------------------------------- ### schedule_maint Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Schedules maintenance for a machine. Requires machine ID, start date, and duration. A category can optionally be provided. ```APIDOC ## schedule_maint ### Description Schedule maintenance for a machine. ### Method Not specified (assumed to be a client-side SDK method) ### Parameters #### Path Parameters * **id** (int) - Required - Machine ID * **sdate** (str) - Required - Start date * **duration** (float) - Required - Duration in hours #### Query Parameters * **category** (str) - Optional - Maintenance category (defaults to "not provided") ### Request Example ```python schedule_maint(id=1, sdate="2024-01-01", duration=4.0, category="GPU check") ``` ### Response #### Success Response - Response dict #### Response Example ```json { "message": "Maintenance scheduled successfully" } ``` ``` -------------------------------- ### SyncClient.create_instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Accepts a specific offer and creates a new instance based on the provided configuration. Requires an offer ID and an InstanceConfig object. ```APIDOC ## SyncClient.create_instance ### Description Accept an offer and create an instance. ### Method POST (implied by creation operation) ### Endpoint /instances (implied base URL + create endpoint) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **offer_id** (int) - Required - Offer ID from search results - **config** (InstanceConfig) - Required - Instance configuration (see types.md) ### Request Example ```python from vastai import SyncClient, InstanceConfig client = SyncClient() config = InstanceConfig( image="pytorch/pytorch:latest", disk=32, label="my-gpu", ) instance = client.create_instance(offer_id=12345, config=config) print(f"Instance created: {instance.id}") ``` ### Response #### Success Response (200) - **instance** (SyncInstance) - SyncInstance wrapper object #### Error Response - **error** (Exception) - Exception if instance creation fails ``` -------------------------------- ### Create Deployment Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Instantiate the Deployment class with a name, local path, and optional version. ```python deployment = Deployment( name="my-inference-service", path="/path/to/code", version="1.0.0", ) ``` -------------------------------- ### async create_instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Asynchronously creates a new instance from a given offer ID and configuration. Returns an AsyncInstance object. ```APIDOC ## async create_instance Create an instance asynchronously. ### Parameters - **offer_id** (int) - Required - The ID of the offer to create the instance from. - **config** (InstanceConfig) - Required - Instance configuration object. **Returns:** AsyncInstance wrapper object **Example:** ```python async def main(): client = AsyncClient() config = InstanceConfig(image="pytorch/pytorch", disk=32) instance = await client.create_instance(12345, config) print(f"Created: {instance.id}") asyncio.run(main()) ``` ``` -------------------------------- ### Get and Use a Specific Endpoint Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SERVERLESS_CLIENT.md Demonstrates how to retrieve a specific endpoint using `get_endpoint` and then make requests through that endpoint. ```python async def main(): serverless = Serverless() endpoint = await serverless.get_endpoint("my-endpoint") response = await endpoint.request( "/v1/completions", { "model": "gpt2", "prompt": "Hello", "max_tokens": 20, } ) print(response) asyncio.run(main()) ``` -------------------------------- ### List All Instances Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Display a list of all your current instances, including their ID, label, GPU type, cost per hour, and status. ```bash vastai show instances ``` -------------------------------- ### Run Worker Server Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/DEPLOYMENT_SERVERSIDE.md Start the worker server to handle incoming requests. This should be run within an asyncio event loop. ```python import asyncio from vastai import Worker, WorkerConfig async def main(): config = WorkerConfig( model_server_url="http://localhost:8000", ) worker = Worker(config) await worker.run() asyncio.run(main()) ``` -------------------------------- ### Initialize BenchmarkConfig Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SERVERLESS_CLIENT.md Demonstrates how to initialize the BenchmarkConfig object with custom dataset, runs, concurrency, and warmup settings. ```python from vastai import BenchmarkConfig benchmark = BenchmarkConfig( dataset=[ {"prompt": "test1"}, {"prompt": "test2"}, ], runs=10, concurrency=5, do_warmup=True, ) ``` -------------------------------- ### Create instance with ComfyUI and environment variables Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Creates an instance using the ComfyUI image, specifying disk size, SSH access, direct connection, and setting checkpoint model and Hugging Face token via environment variables. ```bash vastai create instance --image vastai/comfy:@vastai-automatic-tag --disk 40 --ssh --direct \ --env '-e CHECKPOINT_MODEL=black-forest-labs/FLUX.1-schnell -e HF_TOKEN=hf_xxx' ``` -------------------------------- ### Get Instance SSH URL Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Retrieves the SSH connection URL for an instance, including the username, IP address, and port. ```python ssh_url(id: int) -> str ``` -------------------------------- ### Get SCP Connection URL Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Retrieve the SCP connection URL for a given instance ID, useful for file transfers. ```bash vastai scp-url # Get scp:// URL ``` -------------------------------- ### Get SSL Context Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Retrieves or creates a cached SSL context with the Vast.ai root certificate. This is used for secure communication. ```python async def get_ssl_context() -> ssl.SSLContext ``` -------------------------------- ### Show Help Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Displays general help or command-specific help information for the Vast.ai CLI. ```bash vastai help [COMMAND] # Examples: vastai help # General help vastai help search offers # Command-specific help vastai search offers --help # Also works ``` -------------------------------- ### Initialize VastAI Client with Parameters Source: https://github.com/vast-ai/vast-cli/blob/master/vastai_sdk/SKILL.md Initialize the VastAI client with optional parameters for server URL, retries, raw output, and quiet mode. ```python from vastai import VastAI vast = VastAI(api_key=None, server_url=None, retry=3, raw=False, quiet=False) ``` -------------------------------- ### Get SCP URL for Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Obtain the SCP (Secure Copy Protocol) URL for an instance, used for file transfers. ```bash vastai show-scp-url ID ``` -------------------------------- ### Get Serverless Endpoint by Name Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SERVERLESS_CLIENT.md Retrieve an Endpoint object using its name. This is a prerequisite for making requests to a specific endpoint. ```python serverless = Serverless() endpoint = await serverless.get_endpoint("my-endpoint") ``` -------------------------------- ### SyncClient Constructor Configuration Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CONFIGURATION.md Initialize the SyncClient with API key and server details. ```python from vastai import SyncClient client = SyncClient( api_key="key", # From VAST_API_KEY if omitted vast_server="https://console.vast.ai", ) ``` -------------------------------- ### Get Serverless Endpoint Source: https://github.com/vast-ai/vast-cli/blob/master/README.md Retrieve a serverless endpoint using the initialized client. This endpoint will be used to send inference requests. ```python endpoint = await serverless.get_endpoint("my-endpoint") ``` -------------------------------- ### Make a Simple Completion Request Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SERVERLESS_CLIENT.md Shows how to instantiate the Serverless client and make a basic completion request to a specified model. ```python import asyncio from vastai import Serverless async def main(): serverless = Serverless() response = await serverless.request( "/v1/completions", { "model": "Qwen/Qwen3-8B", "prompt": "What is AI?", "max_tokens": 50, } ) print(response["response"]["choices"][0]["text"]) asyncio.run(main()) ``` -------------------------------- ### Search and Create Instance (Synchronous) Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/INDEX.md Demonstrates a common pattern for searching available offers and creating a new instance using the synchronous VastAI client. ```APIDOC ## Search and Create Instance (Synchronous) ### Description This pattern shows how to search for available machine offers based on specified criteria and then create a new instance from the first matching offer. ### Method ```python from vastai import VastAI from vastai.data.instance import InstanceConfig vast = VastAI() # Search for offers matching the query ooffers = vast.search_offers(query="gpu_name=RTX_4090 num_gpus>=2") # If offers are found, create an instance from the first one if offers: config = InstanceConfig(image="pytorch/pytorch", disk=32) instance = vast.create_instance(offers[0]["id"], **config.__dict__) ``` **Note:** Refer to **[API_REFERENCE.md](API_REFERENCE.md)** for details on `search_offers` and `create_instance`. ``` -------------------------------- ### Create Instances from Multiple Offers Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Iterate through a list of offer IDs and create new instances using a specified Docker image. This is useful for launching multiple similar compute environments. ```bash for id in 12345 12346 12347; do vastai create instance $id --image pytorch/pytorch done ``` -------------------------------- ### Set Vast.ai API Key Source: https://github.com/vast-ai/vast-cli/blob/master/README.md Set your Vast.ai API key using the CLI. This is a required step after installation to authenticate your account. ```bash vastai set api-key YOUR_API_KEY ``` -------------------------------- ### Get Instance SCP URL Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Retrieves the SCP (Secure Copy Protocol) URL for an instance, used for secure file transfers. ```python scp_url(id: int) -> str ``` -------------------------------- ### Instantiate SyncClient Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/SYNC_ASYNC_CLIENTS.md Create an instance of the SyncClient. The API key can be auto-resolved from environment variables or configuration files, or provided explicitly. A custom Vast.ai server URL can also be specified. ```python client = SyncClient() client = SyncClient(api_key="your_api_key") client = SyncClient(vast_server="https://api.vast.ai") ``` -------------------------------- ### Get SSH URL for Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/CLI_COMMANDS.md Retrieve the SSH connection URL for a specific instance, which includes the username, IP address, and port. ```bash vastai show-ssh-url ID ``` -------------------------------- ### Get Raw Dictionary Response Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/ERRORS.md Retrieves the full dictionary response from the show_instances command. Use this when you need the complete, unformatted data. ```python response = vast.show_instances() print(response) # Full dict, not formatted ``` -------------------------------- ### Create a New Instance Source: https://github.com/vast-ai/vast-cli/blob/master/_autodocs/API_REFERENCE.md Create a new instance from a contract offer ID. You can specify the Docker image, disk size, price, label, environment variables, and other options. ```python response = vast.create_instance( id=12345, image="pytorch/pytorch", disk=32, label="my-instance", jupyter_lab=True, ) ``` -------------------------------- ### Get SSH Connection URL Source: https://github.com/vast-ai/vast-cli/blob/master/vastai/SKILL.md Retrieve the SSH connection URL for a given instance ID. This command provides the URL but does not initiate the connection. ```bash vastai ssh-url # Get ssh:// connection URL ```