### Manifest file format example Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows the format of a manifest file: each line is a '/bucket/key' path. Lines starting with '/' are used as-is; other lines are joined with the manifest's directory prefix. ```text /bucket/COCO_YOLO/images/train2017/image001.jpg /bucket/COCO_YOLO/images/train2017/image002.jpg /bucket/COCO_YOLO/images/train2017/image003.jpg ``` -------------------------------- ### Download and install latest version from GitHub Releases Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/installation.md Download the latest osstorchconnector wheel from GitHub Releases and install it with pip. Example uses v1.1.0rc1 for Python 3.11. ```bash wget https://github.com/aliyun/oss-connector-for-ai-ml/releases/download/osstorchconnector%2Fv1.1.0rc1/osstorchconnector-1.1.0rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl pip install osstorchconnector-1.1.0rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl ``` -------------------------------- ### Example manifest file with object name Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows the format of a manifest file containing only object names, one per line. Used to build datasets without listing OSS objects. ```text Img/BadImag/Bmp/Sample001/img001-00001.png Img/BadImag/Bmp/Sample001/img001-00002.png Img/BadImag/Bmp/Sample001/img001-00003.png ``` -------------------------------- ### Example manifest file with object name and label Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows the format of a manifest file with object name and label separated by a space. The label is used by the manifest parser. ```text Img/BadImag/Bmp/Sample001/img001-00001.png label1 Img/BadImag/Bmp/Sample001/img001-00002.png label2 Img/BadImag/Bmp/Sample001/img001-00003.png label3 ``` -------------------------------- ### Download and install OSSModelConnector wheel with wget and pip Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/python_apis.md Downloads the OSSModelConnector wheel for Python 3.11 from the GitHub Release and installs it with pip. The example uses version 1.0.0rc8 and requires Linux x86-64, glibc >= 2.17, Python 3.8-3.12, and PyTorch >= 2.0. ```bash wget https://github.com/aliyun/oss-connector-for-ai-ml/releases/download/ossmodelconnector%2Fv1.0.0rc1/ossmodelconnector-1.0.0rc8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl pip install ossmodelconnector-1.0.0rc8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl ``` -------------------------------- ### Install oss-connector-lib via deb Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/ld_preload.md Install the oss-connector-lib package on Debian-based systems using wget and dpkg. Downloads version 1.0.0rc8 from the official release. ```shell wget https://github.com/aliyun/oss-connector-for-ai-ml/releases/download/ossmodelconnector%2Fv1.0.0rc8/oss-connector-lib-1.0.0rc8.x86_64.deb dpkg -i oss-connector-lib-1.0.0rc8.x86_64.deb ``` -------------------------------- ### OssIterableDataset from_prefix and from_objects usage Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows how to create an OssIterableDataset from a prefix or from a list of object URIs, then iterate over items to read their content. Requires endpoint, config_path, and cred_path parameters. ```py from osstorchconnector import OssIterableDataset ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/" # 1) from_prefix iterable_dataset = OssIterableDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH) for item in iterable_dataset: print(item.key) print(item.size) content = item.read() print(len(content)) item.close() # 2) from_objects uris = [ "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00001.png", "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00002.png", "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00003.png" ] iterable_dataset = OssIterableDataset.from_objects(uris, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)] for item in iterable_dataset: print(item.key) print(item.size) content = item.read() print(len(content)) item.close() ``` -------------------------------- ### Install stable version with pip Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/installation.md Install the stable version of osstorchconnector using pip. ```bash pip install osstorchconnector ``` -------------------------------- ### Create OSS connector configuration file Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/python_apis.md Creates the configuration file at /etc/oss-connector/config.json with settings for logging, prefetch, and fastList. Note: the example JSON is missing a comma after the prefetch block, which may cause parsing errors. ```bash mkdir -p /etc/oss-connector/ cat <<-EOF | tee /etc/oss-connector/config.json { "logLevel": 1, "logPath": "/var/log/oss-connector/connector.log", "auditPath": "/var/log/oss-connector/audit.log", "prefetch": { "vcpus": 24, "workers": 32 } "fastList": { "vcpus": 2, "workers": 16 } } EOF ``` -------------------------------- ### Install oss-connector-lib via rpm Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/ld_preload.md Install the oss-connector-lib package using yum for RPM-based systems. Downloads version 1.0.0rc8 from the official release. ```shell yum install -y https://github.com/aliyun/oss-connector-for-ai-ml/releases/download/ossmodelconnector%2Fv1.0.0rc8/oss-connector-lib-1.0.0rc8.x86_64.rpm ``` -------------------------------- ### OssMapDataset from_prefix and from_objects usage Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows how to create an OssMapDataset from a prefix or from a list of object URIs, supporting random access, context manager usage, and iteration. Note that OssMapDataset performs an OSS list objects operation under the given prefix first, which may take some time. ```py from osstorchconnector import OssMapDataset ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/" # 1) from_prefix map_dataset = OssMapDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH) # random access item = map_dataset[0] print(item.key) content = item.read() print(item.size) print(len(content)) item.close() # or with map_dataset[5] as item: print(item.key) content = item.read() print(item.size) print(len(content)) # iterable for item in map_dataset: print(item.key) print(item.size) content = item.read() print(len(content)) item.close() # 2) from_objects uris = [ "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00001.png", "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00002.png", "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00003.png" ] map_dataset = OssMapDataset.from_objects(uris, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH) # random access item = map_dataset[1] print(item.key) print(item.size) content = item.read() print(len(content)) item.close() # iterable for item in map_dataset: print(item.key) print(item.size) content = item.read() print(len(content)) item.close() ``` -------------------------------- ### Ultralytics training with make_oss_trainer Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Trains a YOLO model using Ultralytics with a custom trainer from osstorchconnector. The trainer resolves OSS URIs, reads images and labels from OSS, and supports both directory-based and manifest file-based dataset configurations. ```py from ultralytics import YOLO from osstorchconnector import make_oss_trainer ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" REGION = "cn-beijing" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_DATA = "coco_oss.yaml" custom_trainer = make_oss_trainer( endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH, ) # Load model model = YOLO("yolo26n.pt") # Train the model results = model.train( trainer=custom_trainer, data=OSS_DATA, epochs=1, batch=16, imgsz=640, fraction=0.005) ``` -------------------------------- ### Load dataset from manifest file with OssIterableDataset.from_manifest_file Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Defines a manifest_parser that reads lines and yields (key, label) tuples, then creates an OssIterableDataset from a local manifest file or one stored on OSS. The parser raises ValueError on format errors. The dataset items expose key, size, label, read(), and close(). ```python from osstorchconnector import OssIterableDataset ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/" # manifest_parser def manifest_parser(reader: io.IOBase) -> Iterable[Tuple[str, str]]: lines = reader.read().decode("utf-8").strip().split("\n") for i, line in enumerate(lines): try: items = line.strip().split(' ') if len(items) >= 2: key = items[0] label = items[1] yield (key, label) elif len(items) == 1: key = items[0] yield (key, '') else: raise ValueError("format error") except ValueError as e: raise e # from local manifest_file iterable_dataset = OssIterableDataset.from_manifest_file("manifest_file", manifest_parser, "oss://ossconnectorbucket/EnglistImg/", endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH) for item in iterable_dataset: print(item.key) print(item.size) print(item.label) content = item.read() print(len(content)) item.close() # manifest_file on oss iterable_dataset = OssIterableDataset.from_manifest_file("oss://ossconnectorbucket/manifest_file/EnglistImg/manifest_file", manifest_parser, "oss://ossconnectorbucket/EnglistImg/", endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH) ``` -------------------------------- ### Manifest file-based dataset configuration YAML Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Defines a YOLO dataset using manifest files that list image paths. This format avoids OSS listing overhead and is suitable for large datasets or repeated loading. The 'train' and 'val' fields point to manifest files. ```yaml path: oss://ossconnectorbucket/COCO_YOLO train: images/train2017.txt val: images/val2017.txt nc: 80 names: 0: person 1: bicycle 2: car # ... (class names continue) ``` -------------------------------- ### Initialize OssModelConnector with endpoint, credentials, and config Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/python_apis.md Creates an OssModelConnector instance using an endpoint, an EnvironmentVariableCredentialsProvider, and a configuration file path. Refer to the configuration documentation for setup details. ```python connector = OssModelConnector(endpoint=ENDPOINT, cred_provider=EnvironmentVariableCredentialsProvider(), config_path='/tmp/config.json') ``` -------------------------------- ### Use OssCheckpoint to read and write checkpoints Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Creates an OssCheckpoint instance with endpoint, credential path, and config path, then uses its reader and writer context managers to load and save PyTorch checkpoints to OSS URIs. In the testing environment, download speed can exceed 15GB/s. ```python import torch from osstorchconnector import OssCheckpoint ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" checkpoint = OssCheckpoint(endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH) # read checkpoint CHECKPOINT_READ_URI = "oss://ossconnectorbucket/checkpoint/epoch.0" with checkpoint.reader(CHECKPOINT_READ_URI) as reader: state_dict = torch.load(reader) # write checkpoint CHECKPOINT_WRITE_URI = "oss://ossconnectorbucket/checkpoint/epoch.1" with checkpoint.writer(CHECKPOINT_WRITE_URI) as writer: torch.save(state_dict, writer) ``` -------------------------------- ### Directory-based dataset configuration YAML Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Defines a YOLO dataset using directory prefixes for train and val images. The connector lists all objects under these prefixes. The 'path' is an OSS URI, and 'train' and 'val' are relative directory prefixes. ```yaml path: oss://ossconnectorbucket/COCO_YOLO train: images/train2017 val: images/val2017 nc: 80 names: 0: person 1: bicycle 2: car # ... (class names continue) ``` -------------------------------- ### Expected OSS object layout Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows the expected directory structure in OSS for the directory-based dataset format. Images and labels are stored under separate 'images' and 'labels' prefixes, with train and val subdirectories. ```text // images/ train2017/ ← ~118k images val2017/ ← 5k images labels/ train2017/ ← YOLO .txt annotations val2017/ ``` -------------------------------- ### Use OssDCPFileSystem for PyTorch distributed checkpoints (DCP) Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Creates an OssDCPFileSystem with endpoint, region, credential path, and config path, then uses its writer and reader to save and load distributed checkpoints via torch.distributed.checkpoint. Supports DCP.save or DCP.async_save; available since v1.2.0rc2. ```python import torchvision import torch.distributed.checkpoint as DCP from osstorchconnector import OssDCPFileSystem import torch ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" REGION = "cn-beijing" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_URI = "oss://ossconnectorbucket/dcp-checkpoint-resnet18" model = torchvision.models.resnet18() # write to OSS fs = OssDCPFileSystem(endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH, region=REGION) oss_storage_writer = fs.writer(OSS_URI) # DCP.save or DCP.async_save checkpoint_future = DCP.async_save( state_dict=model.state_dict(), storage_writer=oss_storage_writer, ) checkpoint_future.result() # load from OSS loaded_state_dict = { key: torch.zeros_like(value) for key, value in model.state_dict().items() } oss_storage_reader = fs.reader(OSS_URI) DCP.load( loaded_state_dict, storage_reader=oss_storage_reader, ) ``` -------------------------------- ### Install pylance for data access Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/osstables-lance-connector/README.md Install the Lance Python library to read and write table data. ```bash pip install pylance ``` -------------------------------- ### Expected OSS object layout for COCO dataset Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows the required directory structure in the OSS bucket for a COCO-format dataset, including annotation files and image folders for training and validation. ```text /COCO/ annotations/ instances_train2017.json instances_val2017.json train2017/ ← training images val2017/ ← validation images ``` -------------------------------- ### Start Python program with LD_PRELOAD Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/ld_preload.md Launch a Python program (e.g., vLLM API server) with OSSModelConnector enabled via LD_PRELOAD. Requires setting ENABLE_CONNECTOR=1 for the entrypoint process and ensuring MODEL_DIR matches the model directory used by the AI framework. ```shell LD_PRELOAD=/usr/local/lib/libossc_preload.so ENABLE_CONNECTOR=1 OSS_ACCESS_KEY_ID=${akid} OSS_ACCESS_KEY_SECRET=${aksecret} OSS_ENDPOINT=${endpoint} OSS_PATH=oss://${bucket}/${path}/ MODEL_DIR=/tmp/model python3 -m vllm.entrypoints.openai.api_server --model /tmp/model --trust-remote-code --tensor-parallel-size 1 --disable-custom-all-reduce ``` -------------------------------- ### Install Python connector with pip Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/osstables-lance-connector/README.md Install the Python connector package. Requires Python 3.9 or later. ```bash pip install --pre osstables-lance-connector ``` -------------------------------- ### Apply transform to dataset with OssIterableDataset.from_prefix Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Defines a transform function that opens image data, applies torchvision transforms (Resize, CenterCrop, ToTensor, Normalize), and returns the transformed tensor and the label. Creates an OssIterableDataset from a prefix and iterates over items, printing the transformed tensor and label. ```python import sys import io import torchvision.transforms as transforms from PIL import Image from osstorchconnector import OssIterableDataset, OssMapDataset ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/" trans = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) def transform(data): try: img = Image.open(io.BytesIO(data.read())).convert('RGB') val = trans(img) except Exception as e: raise e return val, data.label iterable_dataset = OssIterableDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform, cred_path=CRED_PATH, config_path=CONFIG_PATH) for item in iterable_dataset: print(item[0]) print(item[1]) ``` -------------------------------- ### PyTorch DataLoader with OssIterableDataset and OssMapDataset Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Shows how to load OSS data into PyTorch using OssIterableDataset and OssMapDataset. The main DataLoader worker lists objects from OSS and distributes them to other workers, avoiding redundant listing and reading. OssIterableDataset includes prefetch optimization; with multiple workers, iteration order may not be deterministic. Download speed can exceed 10GB/s for many small files. ```py import sys import io import torch import torchvision.transforms as transforms from PIL import Image from osstorchconnector import OssIterableDataset, OssMapDataset ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/" trans = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) def transform(data): try: img = Image.open(io.BytesIO(data.read())).convert('RGB') val = trans(img) except Exception as e: raise e return val, data.key, data.label # OssIterableDataset iterable_dataset = OssIterableDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform, cred_path=CRED_PATH, config_path=CONFIG_PATH) loader = torch.utils.data.DataLoader(iterable_dataset, batch_size=256, num_workers=32, prefetch_factor=2) for i, (datas, keys, labels) in enumerate(loader): print(datas) print(keys) # OssMapDataset with shuffle map_dataset = OssMapDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform, cred_path=CRED_PATH, config_path=CONFIG_PATH) loader = torch.utils.data.DataLoader(map_dataset, batch_size=256, num_workers=32, prefetch_factor=2, shuffle=True) for i, (datas, keys, labels) in enumerate(loader): print(datas) print(keys) ``` -------------------------------- ### MMDetection training with OSSDetDataset and OSSLoadImageFromFile Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Configures an MMDetection training pipeline to load COCO data directly from OSS. It patches the image loader, replaces dataset types, sets OSS connection parameters, caches annotations locally, and runs the MMEngine Runner. Requires an existing MMDetection config file (rtmdet_tiny_8xb32-300e_coco.py) and OSS credentials. ```python import os from mmengine.runner import Runner from mmengine.config import Config # Import osstorchconnector — registers OSSDetDataset + OSSLoadImageFromFile from osstorchconnector import OSSDetDataset, OSSLoadImageFromFile, get_oss_ann_path ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" REGION = "cn-beijing" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_DATA_ROOT = 'oss://ossconnectorbucket/COCO' OSS_TRAIN_ANN = 'annotations/instances_train2017.json' OSS_VAL_ANN = 'annotations/instances_val2017.json' OSS_TRAIN_PREFIX = 'train2017/' OSS_VAL_PREFIX = 'val2017/' OSS_WORK_DIR = './work_dirs/mmdet_oss' MAX_EPOCHS = 1 BATCH_SIZE = 16 NUM_WORKERS = 8 # Load MMDetection config cfg = Config.fromfile('rtmdet_tiny_8xb32-300e_coco.py') # Patch image loader: LoadImageFromFile → OSSLoadImageFromFile def _patch_oss_pipeline(pipeline): out = [] for t in pipeline: t = dict(t) if t.get("type") in ("LoadImageFromFile", "mmdet.LoadImageFromFile"): t = dict(type="OSSLoadImageFromFile") out.append(t) return out cfg.train_dataloader.dataset.pipeline = _patch_oss_pipeline( cfg.train_dataloader.dataset.pipeline) cfg.val_dataloader.dataset.pipeline = _patch_oss_pipeline( cfg.val_dataloader.dataset.pipeline) # Replace dataset type and OSS connection params cfg.merge_from_dict({ "train_dataloader.dataset.type": "OSSDetDataset", "train_dataloader.dataset.data_root": OSS_DATA_ROOT, "train_dataloader.dataset.ann_file": OSS_TRAIN_ANN, "train_dataloader.dataset.data_prefix": dict(img=OSS_TRAIN_PREFIX), "train_dataloader.dataset.oss_endpoint": ENDPOINT, "train_dataloader.dataset.oss_cred_path": CRED_PATH, "train_dataloader.dataset.oss_config_path": CONFIG_PATH, "train_dataloader.dataset.oss_region": REGION, "val_dataloader.dataset.type": "OSSDetDataset", "val_dataloader.dataset.data_root": OSS_DATA_ROOT, "val_dataloader.dataset.ann_file": OSS_VAL_ANN, "val_dataloader.dataset.data_prefix": dict(img=OSS_VAL_PREFIX), "val_dataloader.dataset.oss_endpoint": ENDPOINT, "val_dataloader.dataset.oss_cred_path": CRED_PATH, "val_dataloader.dataset.oss_config_path": CONFIG_PATH, "val_dataloader.dataset.oss_region": REGION, }) # ann_cache_dir: downloaded annotation JSONs live inside work_dir/ann_cache/ ann_cache_dir = os.path.join(OSS_WORK_DIR, "ann_cache") cfg.merge_from_dict({ 'train_dataloader.dataset.ann_cache_dir': ann_cache_dir, 'val_dataloader.dataset.ann_cache_dir': ann_cache_dir, }) # CocoMetric: point at the local cache of the downloaded OSS annotation _, _, _val_local_ann = get_oss_ann_path(OSS_DATA_ROOT, OSS_VAL_ANN, ann_cache_dir) cfg.val_evaluator.ann_file = _val_local_ann cfg.test_evaluator.ann_file = _val_local_ann # Training config overrides cfg.max_epochs = MAX_EPOCHS cfg.train_cfg.max_epochs = MAX_EPOCHS cfg.train_dataloader.batch_size = BATCH_SIZE cfg.train_dataloader.num_workers = NUM_WORKERS cfg.val_dataloader.batch_size = 1 cfg.val_dataloader.num_workers = NUM_WORKERS cfg.work_dir = OSS_WORK_DIR # Train runner = Runner.from_cfg(cfg) runner.train() ``` -------------------------------- ### Add Arrow memory allocator dependency Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/osstables-lance-connector/README.md Arrow is a provided dependency, so add an Arrow allocator to your application. This example uses arrow-memory-netty version 15.0.0 with runtime scope. ```xml org.apache.arrow arrow-memory-netty 15.0.0 runtime ``` -------------------------------- ### Build and Test Commands for Python and Java Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/osstables-lance-connector/README.md Commands to build and test the Python and Java components from source. For Python, install the package in editable mode with dev dependencies and run pytest. For Java, run the test suite with Maven and build the jar with mvn package. ```bash # Python cd python pip install -e ".[dev]" pytest # Java cd java mvn test # run the test suite mvn package # build the jar ``` -------------------------------- ### Save and load safetensors with OssSafetensor Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/examples.md Use OssSafetensor to save tensors to and load tensors from a safetensor file on OSS. Requires endpoint, region, config path, and credential path. Supports saving with metadata, loading with device specification, and reading tensors via safe_open. ```python import torch from osstorchconnector import OssSafetensor ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com" REGION = "cn-beijing" CONFIG_PATH = "/etc/oss-connector/config.json" CRED_PATH = "/root/.alibabacloud/credentials" OSS_URI = "oss://ossconnectorbucket/safetensors/model.safetensors" sfts = OssSafetensor(endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH, region=REGION) # save tensors to safetensor file on OSS tensors = {"embedding": torch.rand((512, 1024)), "attention": torch.rand((256, 256))} metadata = {"a": "a", "b": "b"} sfts.save_file(tensors, OSS_URI, metadata) # load safetensor file from OSS loaded_tensors = sfts.load_file(OSS_URI, device="cpu") # or load tensors by safe_open with sfts.safe_open(OSS_URI, device ="cpu") as f: metadata = f.metadata() # get metadata for key in f.keys(): # read tensors by keys tensor = f.get_tensor(key) ``` -------------------------------- ### Python: Connect to OSS Tables with lance_namespace Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/osstables-lance-connector/README.md Registers the 'osstables' implementation and connects to an OSS Tables endpoint using the lance_namespace API. Requires the osstables_lance_connector package to be imported. The example creates a namespace and lists tables. ```python import lance_namespace import osstables_lance_connector # registers the "osstables" implementation from lance_namespace import CreateNamespaceRequest, ListTablesRequest namespace = lance_namespace.connect( "osstables", { "osstables.uri": "https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance", "osstables.region": "cn-hangzhou", "osstables.access_key_id": "...", "osstables.secret_access_key": "...", }, ) namespace.create_namespace(CreateNamespaceRequest(id=["sales"])) print(namespace.list_tables(ListTablesRequest(id=["sales"])).tables) ``` -------------------------------- ### Java: Connect to OSS Tables with LanceNamespace Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/osstables-lance-connector/README.md Connects to an OSS Tables endpoint using the Java LanceNamespace API. Requires the osstables-lance-connector Java library and Arrow's RootAllocator. The example creates a namespace with id 'sales'. ```java import com.aliyun.lance.osstables.OssTablesNamespace; import java.util.Collections; import java.util.HashMap; import java.util.Map; import org.apache.arrow.memory.RootAllocator; import org.lance.namespace.LanceNamespace; import org.lance.namespace.model.CreateNamespaceRequest; Map options = new HashMap<>(); options.put("osstables.uri", "https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance"); options.put("osstables.region", "cn-hangzhou"); options.put("osstables.access_key_id", "..."); options.put("osstables.secret_access_key", "..."); try (RootAllocator allocator = new RootAllocator()) { LanceNamespace namespace = LanceNamespace.connect(OssTablesNamespace.class.getName(), options, allocator); namespace.createNamespace( new CreateNamespaceRequest().id(Collections.singletonList("sales"))); } ``` -------------------------------- ### Open object in binary or text mode Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/python_apis.md Opens an object via a URI in the format oss://{bucket}/{name}. The second parameter, binary, defaults to True for binary mode; set to False for text mode. After opening, prefetching starts in the order of open calls. ```python # open as binary mode obj = connector.open('oss://ai-testset/dir1/obj1') # open as text mode obj1 = connector.open('oss://ai-testset/dir1/obj1', False) ``` -------------------------------- ### Use OssModelConnector with vLLM Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/framworks.md Initialize OssModelConnector, call prepare_directory to download and prefetch model files from OSS to a local directory, then pass the local directory to vLLM for inference. The connector cleans model_dir before running and downloads smaller files concurrently while prefetching larger files in alphabetical order. Call connector.close() to release resources after inference. ```python from transformers import AutoTokenizer from vllm import LLM, SamplingParams from ossmodelconnector import OssModelConnector # initialize OssModelConnector connector = OssModelConnector(...) # prepare_directory oss_path = "oss://ai-testset/qwen/Qwen25-75B" model_dir = '/root/abc/' connector.prepare_directory(oss_path, model_dir) # pass model_dir to vllm tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) sampling_params = SamplingParams(temperature=0.7, top_p=0.8, repetition_penalty=1.05, max_tokens=512) llm = LLM(model=model_dir, trust_remote_code=True) # close to release resource connector.close() # do inference ``` -------------------------------- ### Use OssModelConnector with Transformers Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/framworks.md Initialize OssModelConnector, call prepare_directory to download and prefetch model files from OSS to a local directory, then pass the local directory to Hugging Face Transformers. The connector cleans model_dir before running and downloads smaller files concurrently while prefetching larger files in alphabetical order. Call connector.close() to release resources after inference. ```python from transformers import AutoModelForCausalLM, AutoTokenizer from ossmodelconnector import OssModelConnector # initialize OssModelConnector connector = OssModelConnector(...) # prepare_directory oss_path = "oss://ai-testset/qwen/Qwen25-75B" model_dir = '/root/abc/' connector.prepare_directory(oss_path, model_dir) # pass model_dir to transformer tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_dir, device_map="cpu", trust_remote_code=True, ).eval() # close to release resource connector.close() # do inference ``` -------------------------------- ### Create JSON credential file Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/configuration.md Creates /root/.alibabacloud/credentials with AccessKeyId, AccessKeySecret, SecurityToken, and Expiration fields. SecurityToken and Expiration are optional; the file must be updated before expiration to avoid authorization errors. ```bash mkdir -p /root/.alibabacloud/ cat <<-EOF | tee /root/.alibabacloud/credentials { "AccessKeyId": "", "AccessKeySecret": "", "SecurityToken": "", "Expiration": "2024-08-02T15:04:05Z" } EOF ``` -------------------------------- ### Initialize OssModelConnector with credentials file Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/python_apis.md Creates an OssModelConnector using a credentials file at /root/.alibabacloud/credentials. The file must be in JSON format and updated before expiration. ```python from ossmodelconnector import OssModelConnector connector = OssModelConnector(endpoint=ENDPOINT, cred_path='/root/.alibabacloud/credentials', config_path='/tmp/config.json') ``` -------------------------------- ### Initialize and configure OssModelConnector in launch.py Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/framworks.md Edit launch.py to import and configure OssModelConnector before calling main(). Uses oss2 with EnvironmentVariableCredentialsProvider and a config file at /etc/connector.json. prepare_directory() loads all models into memory, which may cause crashes with many models; future versions will support lazy loading and garbage collection. ```python from modules import launch_utils import oss2 from oss2.credentials import EnvironmentVariableCredentialsProvider from ossmodelconnector import OssModelConnector ... def main(): ... if __name__ == "__main__": connector = OssModelConnector(endpoint='oss-cn-beijing-internal.aliyuncs.com', cred_provider=EnvironmentVariableCredentialsProvider(), config_path='/etc/connector.json') connector.prepare_directory('oss://ai-testset/Stable-diffusion/', '/root/stable-diffusion-webui/models/Stable-diffusion') main() ``` -------------------------------- ### List model directory contents for Qwen2.5-72B Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/framworks.md Shows the typical model directory layout for Qwen2.5-72B, with 37 large safetensors files and several small files. Use this as a reference for the structure OssModelConnector reads directly from OSS. ```bash # ll -lh /root/Qwen2.5-72B total 136G -rw-r--r-- 1 root root 664 Sep 25 12:23 config.json -rw-r--r-- 1 root root 2 Sep 25 12:23 configuration.json -rw-r--r-- 1 root root 138 Sep 25 12:23 generation_config.json -rw-r--r-- 1 root root 6.8K Sep 25 12:23 LICENSE -rw-r--r-- 1 root root 1.6M Sep 25 12:23 merges.txt -rw-r--r-- 1 root root 3.6G Sep 25 12:28 model-00001-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 12:33 model-00002-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 12:39 model-00003-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 12:44 model-00004-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 12:50 model-00005-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 12:55 model-00006-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 13:00 model-00007-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:06 model-00008-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:11 model-00009-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:17 model-00010-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 13:22 model-00011-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:28 model-00012-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:33 model-00013-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:39 model-00014-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 13:44 model-00015-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:49 model-00016-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 13:55 model-00017-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:00 model-00018-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 14:06 model-00019-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:11 model-00020-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:17 model-00021-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:22 model-00022-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 14:27 model-00023-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:33 model-00024-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:38 model-00025-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:44 model-00026-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 14:49 model-00027-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 14:55 model-00028-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 15:00 model-00029-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 15:05 model-00030-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 15:11 model-00031-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 15:16 model-00032-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 15:22 model-00033-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 15:27 model-00034-of-00037.safetensors -rw-r--r-- 1 root root 3.6G Sep 25 15:32 model-00035-of-00037.safetensors -rw-r--r-- 1 root root 3.8G Sep 25 15:38 model-00036-of-00037.safetensors -rw-r--r-- 1 root root 3.3G Sep 25 15:43 model-00037-of-00037.safetensors -rw-r--r-- 1 root root 78K Sep 25 15:43 model.safetensors.index.json -rw-r--r-- 1 root root 3.8K Sep 25 15:43 README.md -rw-r--r-- 1 root root 7.1K Sep 25 15:43 tokenizer_config.json -rw-r--r-- 1 root root 6.8M Sep 25 15:43 tokenizer.json -rw-r--r-- 1 root root 2.7M Sep 25 15:43 vocab.json ``` -------------------------------- ### Initialization Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/python_apis.md Initializes an OssModelConnector instance with the specified endpoint, credentials provider, and configuration path. ```APIDOC ## Initialization ### Description Initializes an OssModelConnector instance with the specified endpoint, credentials provider, and configuration path. ### Method Constructor ### Endpoint OssModelConnector(endpoint, cred_provider, config_path) ### Parameters - **endpoint** (string) - Required - The OSS endpoint. - **cred_provider** (CredentialsProvider) - Required - The credentials provider, e.g., EnvironmentVariableCredentialsProvider. - **config_path** (string) - Required - Path to the configuration file. ### Request Example ```python connector = OssModelConnector(endpoint=ENDPOINT, cred_provider=EnvironmentVariableCredentialsProvider(), config_path='/tmp/config.json') ``` ``` -------------------------------- ### open(uri, binary=True) Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/modelconnector/python_apis.md Opens an object using a URI in the format oss://{bucket}/{name}. The binary parameter controls whether the file is opened in binary or text mode. Prefetching is initiated in the order of open calls. ```APIDOC ## open(uri, binary=True) ### Description Opens an object using a URI in the format oss://{bucket}/{name}. The binary parameter controls whether the file is opened in binary or text mode. Prefetching is initiated in the order of open calls. ### Method open ### Endpoint connector.open(uri, binary) ### Parameters - **uri** (string) - Required - The object URI in the format oss://{bucket}/{name}. - **binary** (bool) - Optional - If True, opens in binary mode; if False, opens in text mode. Defaults to True. ### Request Example ```python # open as binary mode obj = connector.open('oss://ai-testset/dir1/obj1') # open as text mode obj1 = connector.open('oss://ai-testset/dir1/obj1', False) ``` ``` -------------------------------- ### Initialize OssMapDataset with EnvironmentVariableCredentialsProvider Source: https://github.com/aliyun/oss-connector-for-ai-ml/blob/main/docs/torchconnector/configuration.md Passes the EnvironmentVariableCredentialsProvider to the OssMapDataset constructor to authenticate using the environment variables set above. ```python import oss2 from oss2.credentials import EnvironmentVariableCredentialsProvider from osstorchconnector import OssMapDataset map_dataset = OssMapDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, cred_provider=EnvironmentVariableCredentialsProvider(), config_path=CONFIG_PATH) ```