Tencent Cloud TI-ONE Platform

https://www.tencentcloud.com/document/product/1141
Admin
Tencent Cloud TI-ONE is a one-stop large model training and inference platform that provides AI...
Tokens:97,699
Snippets:1,161
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Create Launch Script

Source: https://www.tencentcloud.com/document/product/1141/69622

Define the entrypoint.sh script to set up the environment, install dependencies, and start the tiinfer service.

```bash
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64

MODEL_DIR=/opt/model

echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}

if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi

echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
```

--------------------------------

### Start vllm inference service

Source: https://www.tencentcloud.com/document/product/1141/69648

Example command to start the vllm OpenAI-compatible API server with lookahead decoding and quantization enabled.

```bash
python3 -m vllm.entrypoints.openai.api_server --model /data/model --served-model-name model --trust-remote-code --quantization ifq --use-v2-block-manager --use-lookahead --num-speculative-tokens 6
```

--------------------------------

### Custom Image Dockerfile Example

Source: https://www.tencentcloud.com/document/product/1141/69666

A comprehensive Dockerfile template for creating custom training and Notebook images, featuring Tencent Cloud mirror configuration, SSH setup, and TCCL RDMA optimization.

```dockerfile
# [Recommended] Use NVIDIA's PyTorch image as the basic image to be compatible with newer open-source libraries and GPU card types.
FROM nvcr.io/nvidia/pytorch:23.07-py3

# [Recommended] Modify the software source (if using in Tencent Cloud, it is recommended to use the private network source).
# [Tencent Public Network Software Source] mirrors.tencent.com
# [Tencent Cloud Private Network Software Source] mirrors.tencentyun.com
ENV TENCENT_MIRRORS="mirrors.tencentyun.com"
RUN sed -i "s/archive.ubuntu.com/${TENCENT_MIRRORS}/g" /etc/apt/sources.list && \
    sed -i "s/security.ubuntu.com/${TENCENT_MIRRORS}/g" /etc/apt/sources.list && \
    pip config set global.index-url http://${TENCENT_MIRRORS}/pypi/simple && \
    pip config set global.no-cache-dir true && \
    pip config set global.trusted-host ${TENCENT_MIRRORS}

# [Recommended] If using NVIDIA's PyTorch mirror, it is recommended to delete the default NVIDIA source to speed up pip package query and installation.
RUN rm /etc/xdg/pip/pip.conf /etc/pip.conf /root/.pip/pip.conf /root/.config/pip/pip.conf && pip config unset global.extra-index-url

# [Basic Image Specification] Install openssh-server. The SSH login functionality of ****notebook and task-based modeling both depend on the openssh-server component.
RUN apt-get update && apt-get install -y openssh-server && apt-get clean && mkdir -p /var/run/sshd

# [Notebook Image Specification] Configure the /opt/dl/run startup entry
RUN mkdir -p /opt/dl && echo "cd /home/tione/notebook && jupyter lab --allow-root --no-browser --ip=0.0.0.0 --port=8888 --notebook-dir=/home/tione/notebook --NotebookApp.allow_origin='*' --NotebookApp.token=''" > /opt/dl/run && chmod a+x /opt/dl/run

# [Recommended] Use tini as the entrypoint to facilitate reclaiming zombie processes
RUN apt-get update && apt-get install -y tini && apt-get clean
ENTRYPOINT ["/usr/bin/tini", "-g", "--"]

# [Optional - Recommended installation when using HCC-GPU instances] TCCL RDMA communication optimization
# (If using NVIDIA's PyTorch mirror, need to delete the pre-installed NCCL plugin in /opt/hpcx/nccl_rdma_sharp_plugin/lib)
RUN wget https://taco-1251783334.cos.ap-shanghai.myqcloud.com/nccl/nccl-rdma-sharp-plugins_1.2_amd64.deb && \
    dpkg -i nccl-rdma-sharp-plugins_1.2_amd64.deb && rm -f nccl-rdma-sharp-plugins_1.2_amd64.deb && \
    rm -rf /opt/hpcx/nccl_rdma_sharp_plugin/lib/*

# [Optional] Install Tikit (excluding big data components)
RUN pip install tencentcloud-sdk-python==3.0.955 coscmd==1.8.6.31 && \
    pip install --no-dependencies -U tikit

# [Custom] Install required dependency libraries.
RUN pip3 install accelerate==0.21.0 bitsandbytes==0.40.2 datasets==2.14.1 deepspeed==0.10.0 evaluate==0.4.0 peft==0.4.0 protobuf==3.20.3 scipy==1.10.1 sentencepiece==0.1.99 transformers==4.31.0
```

--------------------------------

### Full Request Example

Source: https://www.tencentcloud.com/document/product/1141/73403

A complete example of an HTTP request including the Authorization header.

```http
POST https://cvm.tencentcloudapi.com/
Authorization: TC3-HMAC-SHA256 Credential=AKID********************************/2019-02-25/cvm/tc3_request, SignedHeaders=content-type;host, Signature=a7b8551448762bd123d6f79e81815e31a92013640a6cef36a08ad4b292a4d2f2
Content-Type: application/json; charset=utf-8
Host: cvm.tencentcloudapi.com
X-TC-Action: DescribeInstances
X-TC-Version: 2017-03-12
X-TC-Timestamp: 1551113065
X-TC-Region: ap-guangzhou

{"Limit": 1, "Filters": [{"Values": ["unnamed"], "Name": "instance-name"}]}
```

--------------------------------

### View Help for Inference Environment Script

Source: https://www.tencentcloud.com/document/product/1141/69621

Displays usage information and available parameters for the installation script.

```bash
python init_infer_env.py -h
```

--------------------------------

### Example Authorization Header Value

Source: https://www.tencentcloud.com/document/product/1141/73403

A concrete example of the calculated Authorization header value.

```text
TC3-HMAC-SHA256 Credential=AKID********************************/2019-02-25/cvm/tc3_request, SignedHeaders=content-type;host, Signature=a7b8551448762bd123d6f79e81815e31a92013640a6cef36a08ad4b292a4d2f2
```

--------------------------------

### Implement HelloWorld service

Source: https://www.tencentcloud.com/document/product/1141/69623

A basic example demonstrating worker implementation and logging configuration for a model service.

```python
from typing import Dict
import tiinfer

### Log Code ###
# Add the following code to print the request log recorded by the framework to the terminal.
import logging
import sys
logger_formatter = "%(asctime)s %(levelname)s     %(module)s:%(lineno)d     %(message)s"
logging.basicConfig(stream=sys.stdout, format=logger_formatter, level=logging.DEBUG)
### Log Code ###

# tiinfer supports the native mosec.Worker.
class HelloWorld(mosec.Worker):
    def forward(self, req: Dict) -> Dict:
        return {"hello": f"world. raw req is {req}"}

# Launch two processes to process requests simultaneously.
tiinfer.append_worker(HelloWorld, num=2)
```

--------------------------------

### Install tiinfer package

Source: https://www.tencentcloud.com/document/product/1141/69623

Use pip to install the tiinfer package from the Tencent Cloud PyPI mirror.

```bash
pip install -i https://mirrors.cloud.tencent.com/pypi/simple tiinfer
```

--------------------------------

### Install tikit offline

Source: https://www.tencentcloud.com/document/product/1141/69720

Install tikit using a downloaded wheel file or from source code.

```bash
pip3 install tikit-1.0.0-py3-none-any.whl
  

# Or, decompress the source code.
python3 setup.py install
```

--------------------------------

### MPI Hostfile Configuration

Source: https://www.tencentcloud.com/document/product/1141/69605

Example content for the /etc/mpi/hostfile, defining instance domain names and process counts.

```text

train-960258573108964736-7an39bddmfpc-launcher slots=1
train-960258573108964736-7an39bddmfpc-worker-0 slots=1

```

--------------------------------

### tiinfer entrypoint.sh Script

Source: https://www.tencentcloud.com/document/product/1141/69622

The default startup script for the tiinfer framework that installs dependencies and launches the inference service.

```bash
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64


MODEL_DIR=/data/model


echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}


if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi


echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
```

--------------------------------

### Start inference service

Source: https://www.tencentcloud.com/document/product/1141/69623

Command to launch the inference service using the model_service.py file.

```bash
TI_MODEL_DIR=pwd python3 -m tiinfer --timeout 30000
```

--------------------------------

### Start TI-ONE Inference Service

Source: https://www.tencentcloud.com/document/product/1141/69621

Run this command in the directory containing model_service.py to start the inference service with a specified timeout.

```bash
python3 -m tiinfer --timeout 30000
```

--------------------------------

### Dockerfile Base Image Reference

Source: https://www.tencentcloud.com/document/product/1141/69622

Example instruction for referencing the base image in a custom Dockerfile.

```dockerfile
FROM ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0
```

--------------------------------

### HTTP GET Request Structure

Source: https://www.tencentcloud.com/document/product/1141/73402

Example of an HTTP GET request including common parameters in the URL query string.

```http
https://cvm.tencentcloudapi.com/?Action=DescribeInstances&Version=2017-03-12&SignatureMethod=HmacSHA256&Timestamp=1527672334&Signature=37ac2f4fde00b0ac9bd9eadeb459b1bbee224158d66e7ae5fcadb70b2d181d02&Region=ap-guangzhou&Nonce=23823223&SecretId=AKID********************************

Host: cvm.tencentcloudapi.com
Content-Type: application/x-www-form-urlencoded
```

--------------------------------

### Configure Startup Command

Source: https://www.tencentcloud.com/document/product/1141/69605

Command to execute the training script after mounting the code directory.

```bash
cd /opt/ml/code; python job.py
```

--------------------------------

### HTTP GET Request Structure

Source: https://www.tencentcloud.com/document/product/1141/73402

Example of an HTTP GET request for querying CVM instances, with parameters passed in the URL and authentication headers.

```http
https://cvm.tencentcloudapi.com/?Limit=10&Offset=0

Authorization: TC3-HMAC-SHA256 Credential=AKID********************************/2018-10-09/cvm/tc3_request, SignedHeaders=content-type;host, Signature=5da7a33f6993f0614b047e5df4582db9e9bf4672ba50567dba16c6ccf174c474
Content-Type: application/x-www-form-urlencoded
Host: cvm.tencentcloudapi.com
X-TC-Action: DescribeInstances
X-TC-Version: 2017-03-12
X-TC-Timestamp: 1539084154
X-TC-Region: ap-guangzhou
```

--------------------------------

### Distributed Training Startup Commands

Source: https://www.tencentcloud.com/document/product/1141/69605

Commands to initiate training in MPI or Horovod modes using environment variables for node and GPU configuration.

```bash

# Startup in MPI mode
mpirun --allow-run-as-root -np $GPU_NUM -H $NODE_IP_SLOT_LIST python3 train.py --data-dir /opt/ml/input/data

# Startup in Horovod mode
horovodrun -np $GPU_NUM -H $NODE_IP_SLOT_LIST --network-interface eth0 python3 train.py --data-dir /opt/ml/input/data

```

--------------------------------

### Default Startup Command

Source: https://www.tencentcloud.com/document/product/1141/69648

The default command used to initiate the angel-vllm mirror.

```text
run
```

--------------------------------

### Install Inference Environment for PMML

Source: https://www.tencentcloud.com/document/product/1141/69621

Command to install dependencies for PMML models, which requires local OpenJDK 1.8 installation.

```bash
python init_infer_env.py --framework jpmml0.6.2 --model-format PMML
```

--------------------------------

### Example StringToSign

Source: https://www.tencentcloud.com/document/product/1141/73403

A concrete example of the concatenated string to be signed.

```text
TC3-HMAC-SHA256
1551113065
2019-02-25/cvm/tc3_request
2815843035062fffda5fd6f2a44ea8a34818b0dc46f024b8b3786976a3adda7a
```

--------------------------------

### Native vLLM Entrypoint Command

Source: https://www.tencentcloud.com/document/product/1141/69646

The default startup command for the vllm-openai Docker image.

```bash
python3 -m vllm.entrypoints.openai.api_server
```

--------------------------------

### Install Inference Environment for ONNX

Source: https://www.tencentcloud.com/document/product/1141/69621

Command to install dependencies for ONNX model inference.

```bash
python init_infer_env.py --framework onnx1.11.0 --model-format ONNX --model-scene detect --demo-dir ./
```

--------------------------------

### Install tikit via pip

Source: https://www.tencentcloud.com/document/product/1141/69720

Use pip to install or upgrade the tikit package.

```bash
pip3 install -U tikit
```

--------------------------------

### Launch DDP Distributed Training

Source: https://www.tencentcloud.com/document/product/1141/69667

Start training scripts with DDP communication optimization without modifying existing code.

```bash
python3 -u -m tiacc_training.distributed.launch --nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT main.py
```

--------------------------------

### Describe Training Frameworks

Source: https://www.tencentcloud.com/document/product/1141/69714

Lists the available training frameworks supported by the platform.

```python
def describe_training_frameworks(self):
    """view Training framework
  

    :rtype: :class:`tikit.tencentcloud.tione.v20211111.models.DescribeTrainingFrameworksResponse`
  

    """
```

--------------------------------

### Create a Training Task

Source: https://www.tencentcloud.com/document/product/1141/69714

Initializes a new training task with specified resources, framework, and configuration parameters.

```python
def create_training_task(self,
                            name,
                            framework,
                            cos_output,
                            worker_resource,
                            code_package_path,
                            ps_resource=None,
                            input_data_config=None,
                            worker_start_cmd=None,
                            ps_start_cmd=None,
                            tags=None,
                            tuning_parameters_dict={},
                            resource_group_id="",
                            remark=None,
                            log_enable=False,
                            log_logset_id=None,
                            log_topic_id=None,
                            vpc_id=None,
                            sub_net_id=None,
                            restart_limit=0):
    Create training task
  

    :param name:        task name
    :type name:         str
    :param framework:    the running framework environment
    :type framework:    :class:`tikit.models.FrameworkInfo`
    :param cos_output:          output cos information
    :type cos_output:           str
    :param worker_resource:     worker node configuration
    :type worker_resource:      :class:`tikit.models.ResourceConfigInfo`
    :param code_package_path:   cos information of the code
    :type code_package_path:    str
    :param ps_resource:         ps node configuration
    :type ps_resource:          :class:`tikit.models.ResourceConfigInfo`
    :param input_data_config:   input data information
    :type input_data_config:    list or :class:`tikit.models.TrainingDataConfig`
    :param worker_start_cmd:    worker startup command
    :type worker_start_cmd:     str
    :param ps_start_cmd:        ps node startup command
    :type ps_start_cmd:         str
    :param tags:                Tag
    :type tags:                 list of :class:`tikit.tencentcloud.tione.v20211111.models.Tag`
    :param tuning_parameters_dict:  Tuning parameters dictionary
    :type tuning_parameters_dict:   dict
    :param resource_group_id:    prepaid resource group id
    :type resource_group_id:    str
    :param remark:              Description
    :type remark:               str
    :param log_enable:          log switch
    :type log_enable:           bool
    :param log_logset_id:       log set id
    :type log_logset_id:        str
    :param log_topic_id:        log topic id
    :type log_topic_id:         str
    :param vpc_id:              vpc id
    :type vpc_id:               str
    :param sub_net_id:          subnet ID
    :type sub_net_id:           str
    :param restart_limit:       Maximum number of restarts for the current task, max 10, after this limit is exceeded the task is marked as abnormal 
    :type restart_limit:          int
    :return:
    :rtype: :class:`tikit.tencentcloud.tione.v20211111.models.CreateTrainingTaskResponse`
    """
```

--------------------------------

### DescribeModelServiceGroups Response Example

Source: https://www.tencentcloud.com/document/product/1141/73408

Example JSON response containing service group details.

```json
{
    "Response": {
        "ServiceGroups": [
            {
                "ServiceGroupId": "ms-vzd8qdk5",
                "ServiceGroupName": "llm_test",
                "CreatedBy": "100032054859",
                "CreateTime": "2023-08-05T05:46:58Z",
                "UpdateTime": "2023-08-08T14:27:02Z",
                "Uin": "100005348929",
                "ServiceCount": 1,
                "RunningServiceCount": 0,
                "Services": [
                    {
                        "ServiceGroupId": "ms-vzd8qdk5",
                        "ServiceId": "ms-vzd8qdk5-1",
                        "ServiceGroupName": "llm_test",
                        "ServiceDescription": "",
                        "ServiceInfo": {
                            "Replicas": 0,
                            "ImageInfo": {
                                "ImageType": "PRE_SET",
                                "ImageUrl": "tione.tencentcloudcr.com/qcloud-ti-platform/chatglm-demo:latest",
                                "RegistryRegion": "",
                                "RegistryId": ""
                            },
                            "Env": [],
                            "Resources": {
                                "Cpu": 12000,
                                "Memory": 45056,
                                "Gpu": 100,
                                "RealGpu": 100,
                                "GpuType": "A10",
                                "RealGpuDetailSet": [
                                    {
                                        "Name": "A10",
                                        "Value": 100
                                    }
                                ]
                            },
                            "InstanceType": "TI.GNV4.3XLARGE44.POST",
                            "ModelInfo": {
                                "ModelId": "m-838176671237008128",
                                "ModelName": "chatglm-6b-tiacc-ft",
                                "ModelVersionId": "mv-v1-838176671237008129",
                                "ModelVersion": "v1",
                                "ModelSource": "COS",
                                "ModelType": "NORMAL",
                                "CosPathInfo": {
                                    "Bucket": "danerli-guangzhou-1256580188",
                                    "Region": "ap-guangzhou",
                                    "Paths": [
                                        "output/ai_market_algo_test_thu_tiacc_ft/train-793371624977165056/output/adgen-chatglm-6b-ft-1e-8/checkpoint-500/model_service.py"
                                    ]
                                },
                                "AlgorithmFramework": "PYTORCH",
                                "ModelFormat": "PYTORCH"
                            },
                            "InferCodeInfo": {
                                "CosPathInfo": {
                                    "Bucket": "danerli-guangzhou-1256580188",
                                    "Region": "ap-guangzhou",
                                    "Paths": [
                                        "output/ai_market_algo_test_thu_tiacc_ft/train-793371624977165056/output/adgen-chatglm-6b-ft-1e-8/checkpoint-500/model_service.py"
                                    ]
                                }
                            },
                            "VolumeMount": {
                                "VolumeSourceType": "CFS",
                                "CFSConfig": {
                                    "Id": "cfs-mpjk7vit",
                                    "Path": "/",
                                    "MountType": "",
                                    "Protocol": ""
                                }
                            },
                            "LogEnable": false,
                            "LogConfig": null,
                            "AuthorizationEnable": false
```

--------------------------------

### DescribeModelServiceGroups Request Example

Source: https://www.tencentcloud.com/document/product/1141/73408

Example HTTP POST request to list service groups.

```http
POST / HTTP/1.1
Host: tione.intl.tencentcloudapi.com
Content-Type: application/json
X-TC-Action: DescribeModelServiceGroups
<Common request parameters>

{
    "Limit": 10,
    "Offset": 0,
    "Filters": []
}
```

--------------------------------

### Install tikit dependencies

Source: https://www.tencentcloud.com/document/product/1141/69720

Install required SASL libraries on CentOS or Ubuntu before using tikit.

```bash
#centos:
  

sudo yum -y install cyrus-sasl cyrus-sasl-devel cyrus-sasl-lib
  

#ubuntu:
  

sudo apt-get update
sudo apt-get install -y libsasl2-dev
```

--------------------------------

### Dockerfile for Notebook Instances

Source: https://www.tencentcloud.com/document/product/1141/69666

Requires openssh-server, JupyterLab, and a configured /opt/dl/run startup script.

```dockerfile

# Self-modify the basic image
FROM ubuntu:20.04
  

# Install openssh-server
RUN apt-get update && apt-get install -y openssh-server && apt-get clean && mkdir -p /var/run/sshd
  

# Install python3, pip3
RUN apt-get update && apt-get install -y python3.8 python3.8-distutils curl && \
    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.8 get-pip.py && rm -f get-pip.py
  

# Install jupyterlab
RUN pip3 install jupyterlab
  

# Configure the /opt/dl/run startup script
RUN mkdir -p /opt/dl && echo "cd /home/tione/notebook && jupyter lab --allow-root --no-browser --ip=0.0.0.0 --port=8888 --notebook-dir=/home/tione/notebook --NotebookApp.allow_origin='*' --NotebookApp.token=''" > /opt/dl/run && chmod a+x /opt/dl/run

```

--------------------------------

### Install Inference Environment for PyTorch 1.12.1

Source: https://www.tencentcloud.com/document/product/1141/69621

Command to install dependencies for TorchScript models using PyTorch 1.12.1.

```bash
python init_infer_env.py --framework torch1.12.1 --model-format TorchScript --model-scene {detect, classify, ocr, nlp} --demo-dir ./
```

--------------------------------

### Configure Training Frameworks

Source: https://www.tencentcloud.com/document/product/1141/69714

Define custom or built-in training frameworks for tasks. Use describe_training_frameworks() to retrieve valid options.

```python
def new_custom(training_mode, image_type, image_url, registry_region=None, registry_id=None,
                  user_name=None, passwd=None):
      Custom Training framework configuration View list via describe_training_frameworks()

      :param training_mode:   training mode, for example "DDP", only create training task need to configure
      :type training_mode:    str
      :param image_type:      image type, CCR means Tencent Cloud container mirroring service personal edition, TCR refers to TCR Enterprise Edition, CUSTOM indicates external Custom Image
      :type image_type:       str
      :param image_url:       image address required
      :type image_url:        str
      :param registry_region: region of the mirror repository in Tencent Cloud container mirroring service
      :type registry_region:  str
      :param registry_id: ID of the mirror repository in Tencent Cloud container mirroring service
      :type registry_id:      str
      :param user_name:       userName of the Custom Image repository
      :type user_name:       str
      :param passwd:          password of the Custom Image repository
      :type passwd:           str
      :return:
      :rtype:
      """
```

```python
def new_system_framework(framework_name, framework_environment, training_mode):
    """built-in Training framework

    :param framework_name:      framework name. View list via describe_training_frameworks()
    :type framework_name:       str
    :param framework_environment:    framework environment. View list via describe_training_frameworks()
    :type framework_environment:    str
    :param training_mode:       training mode. View the list via describe_training_frameworks()
    :type training_mode:        str
    :return:
    :rtype:
    """
```

--------------------------------

### Example CanonicalRequest String

Source: https://www.tencentcloud.com/document/product/1141/73403

A concrete example of a fully constructed CanonicalRequest string based on the specified concatenation rules.

```text
POST
/

content-type:application/json; charset=utf-8
host:cvm.tencentcloudapi.com

content-type;host
99d58dfbc6745f6747f36bfca17dee5e6881dc0428a0a36f96199342bc5b4907
```

--------------------------------

### Install Inference Environment for TensorFlow

Source: https://www.tencentcloud.com/document/product/1141/69621

Commands to install dependencies for FrozenGraph and SavedModel formats using TensorFlow 2.4.0 or 1.15.0.

```bash
python init_infer_env.py --framework tf2.4.0 --model-format FrozenGraph --model-scene nlp --demo-dir ./
```

```bash
python init_infer_env.py --framework tf2.4.0 --model-format SavedModel --model-scene {nlp, recommend} --demo-dir ./
```

```bash
python init_infer_env.py --framework tf1.15.0 --model-format FrozenGraph --model-scene nlp --demo-dir ./
```

--------------------------------

### Fine-tuning Parameter Suggestions

Source: https://www.tencentcloud.com/document/product/1141/69667

Recommended configuration parameters for fine-tuning Llama3 8B models on A100 and A800 GPU clusters.

```text
# 8 x A100 40G default parameter
GradienAccumulationSteps=64
BatchSize=1
GradientCheckPointing=False
TilearnHybridTPSize=2
TilearnHybridPPSize=2
            
# 8 x A800 80G Default Parameters
GradienAccumulationSteps=32
BatchSize=1
GradientCheckPointing=False
TilearnHybridTPSize=1
TilearnHybridPPSize=2
TilearnHybridZeroStage=1
```

--------------------------------

### Training data directory structure

Source: https://www.tencentcloud.com/document/product/1141/73647

Example layout of a training data directory containing a data file and the configuration file.

```text
/opt/ml/input/data/train
├── dataset1.json
└── dataset_info.json
```

--------------------------------

### Install and Update Tilearn-Angel Packages

Source: https://www.tencentcloud.com/document/product/1141/69667

Commands to uninstall existing versions and install the required tilearn-llm and tilearn.ops packages, along with the colossalai dependency.

```bash
# tilearn-llm>=0.9.3
# tilearn.ops>=0.2.1.172
pip3 uninstall -y tilearn.llm tilearn.ops
pip3 install tilearn-llm==0.9.3 -i https://pypi.tuna.tsinghua.edu.cn/simple 
pip3 install tilearn.ops==0.2.1.172 -i https://g-bnvx3728-pypi.pkg.coding.net/tione/tilearn/simple
wget https://tione-public-cos-1308945662.cos.ap-shanghai.myqcloud.com/tilearn/hybrid_parallel/colossalai-0.3.4.1-cp310-cp310-linux_x86_64.whlpip3 install colossalai-0.3.4.1-cp310-cp310-linux_x86_64.whl
```

--------------------------------

### Install Inference Environment for PyTorch 1.9.0

Source: https://www.tencentcloud.com/document/product/1141/69621

Commands to install dependencies for TorchScript, HuggingFace, Detectron2, and MMDetection models using PyTorch 1.9.0.

```bash
python init_infer_env.py --framework torch1.9.0 --model-format TorchScript --model-scene {detect, classify, ocr, nlp} --demo-dir ./
```

```bash
python init_infer_env.py --framework torch1.9.0 --model-format HuggingFace --model-scene nlp --demo-dir ./
```

```bash
python init_infer_env.py --framework torch1.9.0 --model-format Detectron2 --model-scene detect --demo-dir ./
```

```bash
python init_infer_env.py --framework torch1.9.0 --model-format MMDetection --model-scene detect --demo-dir ./
```
Tokens:
Raw