### Create Launch Script

Source: https://www.tencentcloud.com/document/product/1141/69622

Define the entrypoint.sh script to set up the environment, install dependencies, and start the tiinfer service.

```bash
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64

MODEL_DIR=/opt/model

echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}

if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi

echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
```

--------------------------------

### Start vllm inference service

Source: https://www.tencentcloud.com/document/product/1141/69648

Example command to start the vllm OpenAI-compatible API server with lookahead decoding and quantization enabled.

```bash
python3 -m vllm.entrypoints.openai.api_server --model /data/model --served-model-name model --trust-remote-code --quantization ifq --use-v2-block-manager --use-lookahead --num-speculative-tokens 6
```

--------------------------------

### Custom Image Dockerfile Example

Source: https://www.tencentcloud.com/document/product/1141/69666

A comprehensive Dockerfile template for creating custom training and Notebook images, featuring Tencent Cloud mirror configuration, SSH setup, and TCCL RDMA optimization.

```dockerfile
# [Recommended] Use NVIDIA's PyTorch image as the basic image to be compatible with newer open-source libraries and GPU card types.
FROM nvcr.io/nvidia/pytorch:23.07-py3

# [Recommended] Modify the software source (if using in Tencent Cloud, it is recommended to use the private network source).
# [Tencent Public Network Software Source] mirrors.tencent.com
# [Tencent Cloud Private Network Software Source] mirrors.tencentyun.com
ENV TENCENT_MIRRORS="mirrors.tencentyun.com"
RUN sed -i "s/archive.ubuntu.com/${TENCENT_MIRRORS}/g" /etc/apt/sources.list && \
    sed -i "s/security.ubuntu.com/${TENCENT_MIRRORS}/g" /etc/apt/sources.list && \
    pip config set global.index-url http://${TENCENT_MIRRORS}/pypi/simple && \
    pip config set global.no-cache-dir true && \
    pip config set global.trusted-host ${TENCENT_MIRRORS}

# [Recommended] If using NVIDIA's PyTorch mirror, it is recommended to delete the default NVIDIA source to speed up pip package query and installation.
RUN rm /etc/xdg/pip/pip.conf /etc/pip.conf /root/.pip/pip.conf /root/.config/pip/pip.conf && pip config unset global.extra-index-url

# [Basic Image Specification] Install openssh-server. The SSH login functionality of ****notebook and task-based modeling both depend on the openssh-server component.
RUN apt-get update && apt-get install -y openssh-server && apt-get clean && mkdir -p /var/run/sshd

# [Notebook Image Specification] Configure the /opt/dl/run startup entry
RUN mkdir -p /opt/dl && echo "cd /home/tione/notebook && jupyter lab --allow-root --no-browser --ip=0.0.0.0 --port=8888 --notebook-dir=/home/tione/notebook --NotebookApp.allow_origin='*' --NotebookApp.token=''" > /opt/dl/run && chmod a+x /opt/dl/run

# [Recommended] Use tini as the entrypoint to facilitate reclaiming zombie processes
RUN apt-get update && apt-get install -y tini && apt-get clean
ENTRYPOINT ["/usr/bin/tini", "-g", "--"]

# [Optional - Recommended installation when using HCC-GPU instances] TCCL RDMA communication optimization
# (If using NVIDIA's PyTorch mirror, need to delete the pre-installed NCCL plugin in /opt/hpcx/nccl_rdma_sharp_plugin/lib)
RUN wget https://taco-1251783334.cos.ap-shanghai.myqcloud.com/nccl/nccl-rdma-sharp-plugins_1.2_amd64.deb && \
    dpkg -i nccl-rdma-sharp-plugins_1.2_amd64.deb && rm -f nccl-rdma-sharp-plugins_1.2_amd64.deb && \
    rm -rf /opt/hpcx/nccl_rdma_sharp_plugin/lib/*

# [Optional] Install Tikit (excluding big data components)
RUN pip install tencentcloud-sdk-python==3.0.955 coscmd==1.8.6.31 && \
    pip install --no-dependencies -U tikit

# [Custom] Install required dependency libraries.
RUN pip3 install accelerate==0.21.0 bitsandbytes==0.40.2 datasets==2.14.1 deepspeed==0.10.0 evaluate==0.4.0 peft==0.4.0 protobuf==3.20.3 scipy==1.10.1 sentencepiece==0.1.99 transformers==4.31.0
```

--------------------------------

### Full Request Example

Source: https://www.tencentcloud.com/document/product/1141/73403

A complete example of an HTTP request including the Authorization header.

```http
POST https://cvm.tencentcloudapi.com/
Authorization: TC3-HMAC-SHA256 Credential=AKID********************************/2019-02-25/cvm/tc3_request, SignedHeaders=content-type;host, Signature=a7b8551448762bd123d6f79e81815e31a92013640a6cef36a08ad4b292a4d2f2
Content-Type: application/json; charset=utf-8
Host: cvm.tencentcloudapi.com
X-TC-Action: DescribeInstances
X-TC-Version: 2017-03-12
X-TC-Timestamp: 1551113065
X-TC-Region: ap-guangzhou

{"Limit": 1, "Filters": [{"Values": ["unnamed"], "Name": "instance-name"}]}
```

--------------------------------

### View Help for Inference Environment Script

Source: https://www.tencentcloud.com/document/product/1141/69621

Displays usage information and available parameters for the installation script.

```bash
python init_infer_env.py -h
```

--------------------------------

### Example Authorization Header Value

Source: https://www.tencentcloud.com/document/product/1141/73403

A concrete example of the calculated Authorization header value.

```text
TC3-HMAC-SHA256 Credential=AKID********************************/2019-02-25/cvm/tc3_request, SignedHeaders=content-type;host, Signature=a7b8551448762bd123d6f79e81815e31a92013640a6cef36a08ad4b292a4d2f2
```

--------------------------------

### Implement HelloWorld service

Source: https://www.tencentcloud.com/document/product/1141/69623

A basic example demonstrating worker implementation and logging configuration for a model service.

```python
from typing import Dict
import tiinfer

### Log Code ###
# Add the following code to print the request log recorded by the framework to the terminal.
import logging
import sys
logger_formatter = "%(asctime)s %(levelname)s     %(module)s:%(lineno)d     %(message)s"
logging.basicConfig(stream=sys.stdout, format=logger_formatter, level=logging.DEBUG)
### Log Code ###

# tiinfer supports the native mosec.Worker.
class HelloWorld(mosec.Worker):
    def forward(self, req: Dict) -> Dict:
        return {"hello": f"world. raw req is {req}"}

# Launch two processes to process requests simultaneously.
tiinfer.append_worker(HelloWorld, num=2)
```

--------------------------------

### Install tiinfer package

Source: https://www.tencentcloud.com/document/product/1141/69623

Use pip to install the tiinfer package from the Tencent Cloud PyPI mirror.

```bash
pip install -i https://mirrors.cloud.tencent.com/pypi/simple tiinfer
```

--------------------------------

### Install tikit offline

Source: https://www.tencentcloud.com/document/product/1141/69720

Install tikit using a downloaded wheel file or from source code.

```bash
pip3 install tikit-1.0.0-py3-none-any.whl
  

# Or, decompress the source code.
python3 setup.py install
```

--------------------------------

### HTTP GET Request Structure

Source: https://www.tencentcloud.com/document/product/1141/73402

Example of an HTTP GET request including common parameters in the URL query string.

```http
https://cvm.tencentcloudapi.com/?Action=DescribeInstances&Version=2017-03-12&SignatureMethod=HmacSHA256&Timestamp=1527672334&Signature=37ac2f4fde00b0ac9bd9eadeb459b1bbee224158d66e7ae5fcadb70b2d181d02&Region=ap-guangzhou&Nonce=23823223&SecretId=AKID********************************

Host: cvm.tencentcloudapi.com
Content-Type: application/x-www-form-urlencoded
```

--------------------------------

### Example StringToSign

Source: https://www.tencentcloud.com/document/product/1141/73403

A concrete example of the concatenated string to be signed.

```text
TC3-HMAC-SHA256
1551113065
2019-02-25/cvm/tc3_request
2815843035062fffda5fd6f2a44ea8a34818b0dc46f024b8b3786976a3adda7a
```

--------------------------------

### MPI Hostfile Configuration

Source: https://www.tencentcloud.com/document/product/1141/69605

Example content for the /etc/mpi/hostfile, defining instance domain names and process counts.

```text

train-960258573108964736-7an39bddmfpc-launcher slots=1
train-960258573108964736-7an39bddmfpc-worker-0 slots=1

```

--------------------------------

### Install Inference Environment for PMML

Source: https://www.tencentcloud.com/document/product/1141/69621

Command to install dependencies for PMML models, which requires local OpenJDK 1.8 installation.

```bash
python init_infer_env.py --framework jpmml0.6.2 --model-format PMML
```

--------------------------------

### tiinfer entrypoint.sh Script

Source: https://www.tencentcloud.com/document/product/1141/69622

The default startup script for the tiinfer framework that installs dependencies and launches the inference service.

```bash
#!/bin/bash
source /etc/profile
source /root/.bashrc
export LD_LIBRARY_PATH=/usr/local/python3/lib/python3.8/site-packages/torch/lib:/usr/local/openmpi/lib:/usr/local/nccl/lib:/usr/local/cuda/lib64:/usr/local/python3/lib:/usr/local/python3/lib64:/usr/local/openmpi/lib:/usr/local/gcc/lib:/usr/local/gcc/lib64


MODEL_DIR=/data/model


echo "================== code path ${MODEL_DIR}=========="
cd ${MODEL_DIR}


if [ -f "requirements.txt" ]; then
  echo "============== install python requirements ======================"
  echo "python3 -m pip install -r requirements.txt"
  python3 -m pip install -r requirements.txt
  echo "============== install python requirements done ================="
fi


echo "====================== start serving ============================"
echo "python3 -m tiinfer"
export TI_MODEL_DIR=${MODEL_DIR}
python3 -m tiinfer --timeout 30000
```

--------------------------------

### Install tikit via pip

Source: https://www.tencentcloud.com/document/product/1141/69720

Use pip to install or upgrade the tikit package.

```bash
pip3 install -U tikit
```

--------------------------------

### HTTP GET Request Structure

Source: https://www.tencentcloud.com/document/product/1141/73402

Example of an HTTP GET request for querying CVM instances, with parameters passed in the URL and authentication headers.

```http
https://cvm.tencentcloudapi.com/?Limit=10&Offset=0

Authorization: TC3-HMAC-SHA256 Credential=AKID********************************/2018-10-09/cvm/tc3_request, SignedHeaders=content-type;host, Signature=5da7a33f6993f0614b047e5df4582db9e9bf4672ba50567dba16c6ccf174c474
Content-Type: application/x-www-form-urlencoded
Host: cvm.tencentcloudapi.com
X-TC-Action: DescribeInstances
X-TC-Version: 2017-03-12
X-TC-Timestamp: 1539084154
X-TC-Region: ap-guangzhou
```

--------------------------------

### Launch DDP Distributed Training

Source: https://www.tencentcloud.com/document/product/1141/69667

Start training scripts with DDP communication optimization without modifying existing code.

```bash
python3 -u -m tiacc_training.distributed.launch --nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT main.py
```

--------------------------------

### Start inference service

Source: https://www.tencentcloud.com/document/product/1141/69623

Command to launch the inference service using the model_service.py file.

```bash
TI_MODEL_DIR=pwd python3 -m tiinfer --timeout 30000
```

--------------------------------

### Start TI-ONE Inference Service

Source: https://www.tencentcloud.com/document/product/1141/69621

Run this command in the directory containing model_service.py to start the inference service with a specified timeout.

```bash
python3 -m tiinfer --timeout 30000
```

--------------------------------

### DescribeModelServiceGroups Request Example

Source: https://www.tencentcloud.com/document/product/1141/73408

Example HTTP POST request to list service groups.

```http
POST / HTTP/1.1
Host: tione.intl.tencentcloudapi.com
Content-Type: application/json
X-TC-Action: DescribeModelServiceGroups
<Common request parameters>

{
    "Limit": 10,
    "Offset": 0,
    "Filters": []
}
```

--------------------------------

### Install Inference Environment for ONNX

Source: https://www.tencentcloud.com/document/product/1141/69621

Command to install dependencies for ONNX model inference.

```bash
python init_infer_env.py --framework onnx1.11.0 --model-format ONNX --model-scene detect --demo-dir ./
```

--------------------------------

### Install tikit dependencies

Source: https://www.tencentcloud.com/document/product/1141/69720

Install required SASL libraries on CentOS or Ubuntu before using tikit.

```bash
#centos:
  

sudo yum -y install cyrus-sasl cyrus-sasl-devel cyrus-sasl-lib
  

#ubuntu:
  

sudo apt-get update
sudo apt-get install -y libsasl2-dev
```

--------------------------------

### Training data directory structure

Source: https://www.tencentcloud.com/document/product/1141/73647

Example layout of a training data directory containing a data file and the configuration file.

```text
/opt/ml/input/data/train
├── dataset1.json
└── dataset_info.json
```

--------------------------------

### Dockerfile Base Image Reference

Source: https://www.tencentcloud.com/document/product/1141/69622

Example instruction for referencing the base image in a custom Dockerfile.

```dockerfile
FROM ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0
```

--------------------------------

### Example CanonicalRequest String

Source: https://www.tencentcloud.com/document/product/1141/73403

A concrete example of a fully constructed CanonicalRequest string based on the specified concatenation rules.

```text
POST
/

content-type:application/json; charset=utf-8
host:cvm.tencentcloudapi.com

content-type;host
99d58dfbc6745f6747f36bfca17dee5e6881dc0428a0a36f96199342bc5b4907
```

--------------------------------

### Dataset Example Ending with Tool Call

Source: https://www.tencentcloud.com/document/product/1141/73647

A JSON dataset sample demonstrating a dialogue that concludes with a tool call request.

```json
{"messages": [{"role": "system", "content": "You are TIONE's intelligent Q&A bot, capable of understanding user questions and providing fluent answers"}, {"role": "user", "content": "/no_think Please answer the following question: Xiao Li is 25 this year, and Xiao Wang is 20. What is the sum of their ages"}, {"role": "tool_call", "content": "<think>\n\n</think>\n<answer>\n{\"name\":\"calculate_age_sum\", \"arguments\": {\"age_1\": 25, \"age_2\": 20}}\n</answer>"}], "tools": "[{\"name\": \"calculate_age_sum\", \"description\": \"Calculate and return the sum of two people's ages\", \"parameters\": {\"type\": \"object\", \"properties\": {\"age_1\": {\"type\": \"number\", \"description\": \"The first person's age\"}, \"age_2\": {\"type\": \"number\", \"description\": \"The second person's age\"}}, \"required\": [\"age_1\", \"age_2\"]}}]"}
```

--------------------------------

### Install Inference Environment for PyTorch 1.12.1

Source: https://www.tencentcloud.com/document/product/1141/69621

Command to install dependencies for TorchScript models using PyTorch 1.12.1.

```bash
python init_infer_env.py --framework torch1.12.1 --model-format TorchScript --model-scene {detect, classify, ocr, nlp} --demo-dir ./
```

--------------------------------

### Dataset Example Ending with Assistant

Source: https://www.tencentcloud.com/document/product/1141/73647

A complete JSON dataset sample demonstrating a multi-round dialogue that concludes with an assistant response.

```json
{"messages": [{"role": "system", "content": "You are TIONE's intelligent Q&A bot, capable of understanding user questions and providing fluent answers"}, {"role": "user", "content": "Please answer the following question: Xiao Li is 25 this year, and Xiao Wang is 20. What is the sum of their ages"}, {"role": "tool_call", "content": "{\"name\":\"calculate_age_sum\", \"arguments\": {\"age_1\": 25, \"age_2\": 20}}"}, {"role": "tool", "content": "/no_think{\"age_sum\": 45}"}, {"role": "assistant", "content": "<think>\n\n</think>\n<answer>\nAccording to my calculation, the sum of Xiao Li and Xiao Wang's ages is 45\n</answer>"}], "tools": "[{\"name\": \"calculate_age_sum\", \"description\": \"Calculate and return the sum of two people's ages\", \"parameters\": {\"type\": \"object\", \"properties\": {\"age_1\": {\"type\": \"number\", \"description\": \"The first person's age\"}, \"age_2\": {\"type\": \"number\", \"description\": \"The second person's age\"}}, \"required\": [\"age_1\", \"age_2\"]}}]"}
```

--------------------------------

### Basic Dockerfile for Task-based Modeling

Source: https://www.tencentcloud.com/document/product/1141/69666

Requires the openssh-server component to be installed to enable task-based modeling.

```dockerfile

# Self-modify the basic image
FROM ubuntu:20.04
  

# Install openssh-server 
RUN apt-get update && apt-get install -y openssh-server && apt-get clean && mkdir -p /var/run/sshd

```

--------------------------------

### Install Inference Environment for PyTorch 1.9.0

Source: https://www.tencentcloud.com/document/product/1141/69621

Commands to install dependencies for TorchScript, HuggingFace, Detectron2, and MMDetection models using PyTorch 1.9.0.

```bash
python init_infer_env.py --framework torch1.9.0 --model-format TorchScript --model-scene {detect, classify, ocr, nlp} --demo-dir ./
```

```bash
python init_infer_env.py --framework torch1.9.0 --model-format HuggingFace --model-scene nlp --demo-dir ./
```

```bash
python init_infer_env.py --framework torch1.9.0 --model-format Detectron2 --model-scene detect --demo-dir ./
```

```bash
python init_infer_env.py --framework torch1.9.0 --model-format MMDetection --model-scene detect --demo-dir ./
```

--------------------------------

### Configure Resource Rules JSON

Source: https://www.tencentcloud.com/document/product/1141/69708

Example JSON structure for defining resource rules including priority, queuing policy, and preemption settings.

```json
{
  "Version": "1.0",
  "ResourceRule": {
    "DefaultPriority": 0,
    "DefaultQueue": 0,
    "Preempted": 1
  },
  "TaskRules": []
}
```

--------------------------------

### Initialize Models with Tilearn-Angel

Source: https://www.tencentcloud.com/document/product/1141/69667

Examples of importing and initializing models using Tilearn-Angel's transformer wrappers, which maintain compatibility with standard Hugging Face APIs.

```python
### TILEARN.LLM
from tilearn.llm.transformers import LlamaForCausalLM

### The model API is consistent with the standard huggingface
model = LlamaForCausalLM.from_pretrained(...)
```

```python
### TILEARN.LLM
from tilearn.llm.transformers import AutoModelForCausalLM

### The model API is consistent with the standard huggingface
model = AutoModelForCausalLM.from_pretrained(...)
```

--------------------------------

### Start DDP Distributed Training Command

Source: https://www.tencentcloud.com/document/product/1141/69605

Standard command to launch a PyTorch DDP training task using platform-injected environment variables.

```bash
python -m torch.distributed.launch --nproc_per_node $GPU_NUM_PER_NODE --nnodes $WORLD_SIZE --node_rank $RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT
```

--------------------------------

### Training data dataset1.json format

Source: https://www.tencentcloud.com/document/product/1141/73647

Example of a JSON file containing training messages in the sharegpt format.

```json
[
    {
        "messages": [
            {
                "role": "system", 
                "content": "You are the TIONE intelligent Q&A robot, capable of understanding user questions and providing fluent answers"
            }, 
            {
                "role": "user", 
                "content": "Read the following text and summarize it in one sentence.\nText: (CNN) - A federal judge ruled in favor of a former UCLA basketball star on Friday, who sued to terminate NCAA's control over college athletes' name, image, and portrait rights."
            }, 
            {
                "role": "assistant", 
                "content": "The judge supported a former UCLA basketball star who challenged NCAA's information control."
            }
        ]
    }, 
    {
        "messages": [
            {
                "role": "system", 
                "content": "You are the TIONE intelligent Q&A robot, capable of understanding user questions and providing fluent answers"
            }, 
            {
                "role": "user", 
                "content": "Read the following text and summarize it in one sentence.\nText: President Obama apologized on Thursday to Americans who lost their insurance plans due to the federal health law he supported, despite his repeated claims that they could retain their coverage if they wished."
            },
            {
                "role": "assistant", 
                "content": "Obama expressed regret in an interview with NBC about the cancellation of the insurance plan."
            }
        ]
    }
]
```

--------------------------------

### Implement Multi-Stage Inference Pipeline

Source: https://www.tencentcloud.com/document/product/1141/69623

Example of separating pre-processing, inference, and post-processing into distinct worker classes and orchestrating them.

```python

import tiinfer
from mosec import Worker
from typing import Dict, Any
  

  

class MyPreprocess(Worker):
    def forward(self, data: Dict) -> Any:
      # Input is a Dict converted from JSON and requires some necessary pre-processing.
  

      
class MyPredict(Worker):
    def __int__(self):
      super.__init__()
      # Read and load the model.
      
    def forward(self, data: Any) -> Any:
      # The input is the result of pre-processing. The inference result is obtained after inference by calling the model.
  

  

class MyPostprocess(Worker):
    def forward(self, data: Any) -> Dict:
      # The input is the inference result. Convert it into a Dict through post-processing and return it as JSON to the caller.
      
  

# Orchestration handling process: 1 x pre-processing worker > 2 x inference workers > 1 x post-processing worker
tiinfer.append_worker(MyPreprocess, 1)
tiinfer.append_worker(MyPredict, 2)
tiinfer.append_worker(MyPostprocess, 1)


```

--------------------------------

### Initiate Public Network API Request

Source: https://www.tencentcloud.com/document/product/1141/69619

Example of a curl command to call an online service via the API gateway. Include the Authorization header only if authentication is enabled.

```bash
curl -X POST https://ms-9******j-100******5.gw.ap-guangzhou.ti.tencentcs.com/ms-9******j/v1/models/m:predict -H 'Authorization: P*********pg' -H 'Content-Type: application/json' -d ''
```

--------------------------------

### Configure Startup Command

Source: https://www.tencentcloud.com/document/product/1141/69605

Command to execute the training script after mounting the code directory.

```bash
cd /opt/ml/code; python job.py
```

--------------------------------

### Distributed Training Startup Commands

Source: https://www.tencentcloud.com/document/product/1141/69605

Commands to initiate training in MPI or Horovod modes using environment variables for node and GPU configuration.

```bash

# Startup in MPI mode
mpirun --allow-run-as-root -np $GPU_NUM -H $NODE_IP_SLOT_LIST python3 train.py --data-dir /opt/ml/input/data

# Startup in Horovod mode
horovodrun -np $GPU_NUM -H $NODE_IP_SLOT_LIST --network-interface eth0 python3 train.py --data-dir /opt/ml/input/data

```

--------------------------------

### Default Startup Command

Source: https://www.tencentcloud.com/document/product/1141/69648

The default command used to initiate the angel-vllm mirror.

```text
run
```

--------------------------------

### Native vLLM Entrypoint Command

Source: https://www.tencentcloud.com/document/product/1141/69646

The default startup command for the vllm-openai Docker image.

```bash
python3 -m vllm.entrypoints.openai.api_server
```

--------------------------------

### Describe Training Frameworks

Source: https://www.tencentcloud.com/document/product/1141/69714

Lists the available training frameworks supported by the platform.

```python
def describe_training_frameworks(self):
    """view Training framework
  

    :rtype: :class:`tikit.tencentcloud.tione.v20211111.models.DescribeTrainingFrameworksResponse`
  

    """
```

--------------------------------

### DescribeModelServiceGroups Response Example

Source: https://www.tencentcloud.com/document/product/1141/73408

Example JSON response containing service group details.

```json
{
    "Response": {
        "ServiceGroups": [
            {
                "ServiceGroupId": "ms-vzd8qdk5",
                "ServiceGroupName": "llm_test",
                "CreatedBy": "100032054859",
                "CreateTime": "2023-08-05T05:46:58Z",
                "UpdateTime": "2023-08-08T14:27:02Z",
                "Uin": "100005348929",
                "ServiceCount": 1,
                "RunningServiceCount": 0,
                "Services": [
                    {
                        "ServiceGroupId": "ms-vzd8qdk5",
                        "ServiceId": "ms-vzd8qdk5-1",
                        "ServiceGroupName": "llm_test",
                        "ServiceDescription": "",
                        "ServiceInfo": {
                            "Replicas": 0,
                            "ImageInfo": {
                                "ImageType": "PRE_SET",
                                "ImageUrl": "tione.tencentcloudcr.com/qcloud-ti-platform/chatglm-demo:latest",
                                "RegistryRegion": "",
                                "RegistryId": ""
                            },
                            "Env": [],
                            "Resources": {
                                "Cpu": 12000,
                                "Memory": 45056,
                                "Gpu": 100,
                                "RealGpu": 100,
                                "GpuType": "A10",
                                "RealGpuDetailSet": [
                                    {
                                        "Name": "A10",
                                        "Value": 100
                                    }
                                ]
                            },
                            "InstanceType": "TI.GNV4.3XLARGE44.POST",
                            "ModelInfo": {
                                "ModelId": "m-838176671237008128",
                                "ModelName": "chatglm-6b-tiacc-ft",
                                "ModelVersionId": "mv-v1-838176671237008129",
                                "ModelVersion": "v1",
                                "ModelSource": "COS",
                                "ModelType": "NORMAL",
                                "CosPathInfo": {
                                    "Bucket": "danerli-guangzhou-1256580188",
                                    "Region": "ap-guangzhou",
                                    "Paths": [
                                        "output/ai_market_algo_test_thu_tiacc_ft/train-793371624977165056/output/adgen-chatglm-6b-ft-1e-8/checkpoint-500/model_service.py"
                                    ]
                                },
                                "AlgorithmFramework": "PYTORCH",
                                "ModelFormat": "PYTORCH"
                            },
                            "InferCodeInfo": {
                                "CosPathInfo": {
                                    "Bucket": "danerli-guangzhou-1256580188",
                                    "Region": "ap-guangzhou",
                                    "Paths": [
                                        "output/ai_market_algo_test_thu_tiacc_ft/train-793371624977165056/output/adgen-chatglm-6b-ft-1e-8/checkpoint-500/model_service.py"
                                    ]
                                }
                            },
                            "VolumeMount": {
                                "VolumeSourceType": "CFS",
                                "CFSConfig": {
                                    "Id": "cfs-mpjk7vit",
                                    "Path": "/",
                                    "MountType": "",
                                    "Protocol": ""
                                }
                            },
                            "LogEnable": false,
                            "LogConfig": null,
                            "AuthorizationEnable": false
```

--------------------------------

### Create a Training Task

Source: https://www.tencentcloud.com/document/product/1141/69714

Initializes a new training task with specified resources, framework, and configuration parameters.

```python
def create_training_task(self,
                            name,
                            framework,
                            cos_output,
                            worker_resource,
                            code_package_path,
                            ps_resource=None,
                            input_data_config=None,
                            worker_start_cmd=None,
                            ps_start_cmd=None,
                            tags=None,
                            tuning_parameters_dict={},
                            resource_group_id="",
                            remark=None,
                            log_enable=False,
                            log_logset_id=None,
                            log_topic_id=None,
                            vpc_id=None,
                            sub_net_id=None,
                            restart_limit=0):
    Create training task
  

    :param name:        task name
    :type name:         str
    :param framework:    the running framework environment
    :type framework:    :class:`tikit.models.FrameworkInfo`
    :param cos_output:          output cos information
    :type cos_output:           str
    :param worker_resource:     worker node configuration
    :type worker_resource:      :class:`tikit.models.ResourceConfigInfo`
    :param code_package_path:   cos information of the code
    :type code_package_path:    str
    :param ps_resource:         ps node configuration
    :type ps_resource:          :class:`tikit.models.ResourceConfigInfo`
    :param input_data_config:   input data information
    :type input_data_config:    list or :class:`tikit.models.TrainingDataConfig`
    :param worker_start_cmd:    worker startup command
    :type worker_start_cmd:     str
    :param ps_start_cmd:        ps node startup command
    :type ps_start_cmd:         str
    :param tags:                Tag
    :type tags:                 list of :class:`tikit.tencentcloud.tione.v20211111.models.Tag`
    :param tuning_parameters_dict:  Tuning parameters dictionary
    :type tuning_parameters_dict:   dict
    :param resource_group_id:    prepaid resource group id
    :type resource_group_id:    str
    :param remark:              Description
    :type remark:               str
    :param log_enable:          log switch
    :type log_enable:           bool
    :param log_logset_id:       log set id
    :type log_logset_id:        str
    :param log_topic_id:        log topic id
    :type log_topic_id:         str
    :param vpc_id:              vpc id
    :type vpc_id:               str
    :param sub_net_id:          subnet ID
    :type sub_net_id:           str
    :param restart_limit:       Maximum number of restarts for the current task, max 10, after this limit is exceeded the task is marked as abnormal 
    :type restart_limit:          int
    :return:
    :rtype: :class:`tikit.tencentcloud.tione.v20211111.models.CreateTrainingTaskResponse`
    """
```

--------------------------------

### Dockerfile for Notebook Instances

Source: https://www.tencentcloud.com/document/product/1141/69666

Requires openssh-server, JupyterLab, and a configured /opt/dl/run startup script.

```dockerfile

# Self-modify the basic image
FROM ubuntu:20.04
  

# Install openssh-server
RUN apt-get update && apt-get install -y openssh-server && apt-get clean && mkdir -p /var/run/sshd
  

# Install python3, pip3
RUN apt-get update && apt-get install -y python3.8 python3.8-distutils curl && \
    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.8 get-pip.py && rm -f get-pip.py
  

# Install jupyterlab
RUN pip3 install jupyterlab
  

# Configure the /opt/dl/run startup script
RUN mkdir -p /opt/dl && echo "cd /home/tione/notebook && jupyter lab --allow-root --no-browser --ip=0.0.0.0 --port=8888 --notebook-dir=/home/tione/notebook --NotebookApp.allow_origin='*' --NotebookApp.token=''" > /opt/dl/run && chmod a+x /opt/dl/run

```

--------------------------------

### Configure Training Frameworks

Source: https://www.tencentcloud.com/document/product/1141/69714

Define custom or built-in training frameworks for tasks. Use describe_training_frameworks() to retrieve valid options.

```python
def new_custom(training_mode, image_type, image_url, registry_region=None, registry_id=None,
                  user_name=None, passwd=None):
      Custom Training framework configuration View list via describe_training_frameworks()

      :param training_mode:   training mode, for example "DDP", only create training task need to configure
      :type training_mode:    str
      :param image_type:      image type, CCR means Tencent Cloud container mirroring service personal edition, TCR refers to TCR Enterprise Edition, CUSTOM indicates external Custom Image
      :type image_type:       str
      :param image_url:       image address required
      :type image_url:        str
      :param registry_region: region of the mirror repository in Tencent Cloud container mirroring service
      :type registry_region:  str
      :param registry_id: ID of the mirror repository in Tencent Cloud container mirroring service
      :type registry_id:      str
      :param user_name:       userName of the Custom Image repository
      :type user_name:       str
      :param passwd:          password of the Custom Image repository
      :type passwd:           str
      :return:
      :rtype:
      """
```

```python
def new_system_framework(framework_name, framework_environment, training_mode):
    """built-in Training framework

    :param framework_name:      framework name. View list via describe_training_frameworks()
    :type framework_name:       str
    :param framework_environment:    framework environment. View list via describe_training_frameworks()
    :type framework_environment:    str
    :param training_mode:       training mode. View the list via describe_training_frameworks()
    :type training_mode:        str
    :return:
    :rtype:
    """
```

--------------------------------

### Install Inference Environment for TensorFlow

Source: https://www.tencentcloud.com/document/product/1141/69621

Commands to install dependencies for FrozenGraph and SavedModel formats using TensorFlow 2.4.0 or 1.15.0.

```bash
python init_infer_env.py --framework tf2.4.0 --model-format FrozenGraph --model-scene nlp --demo-dir ./
```

```bash
python init_infer_env.py --framework tf2.4.0 --model-format SavedModel --model-scene {nlp, recommend} --demo-dir ./
```

```bash
python init_infer_env.py --framework tf1.15.0 --model-format FrozenGraph --model-scene nlp --demo-dir ./
```

--------------------------------

### Fine-tuning Parameter Suggestions

Source: https://www.tencentcloud.com/document/product/1141/69667

Recommended configuration parameters for fine-tuning Llama3 8B models on A100 and A800 GPU clusters.

```text
# 8 x A100 40G default parameter
GradienAccumulationSteps=64
BatchSize=1
GradientCheckPointing=False
TilearnHybridTPSize=2
TilearnHybridPPSize=2
            
# 8 x A800 80G Default Parameters
GradienAccumulationSteps=32
BatchSize=1
GradientCheckPointing=False
TilearnHybridTPSize=1
TilearnHybridPPSize=2
TilearnHybridZeroStage=1
```

--------------------------------

### Install and Update Tilearn-Angel Packages

Source: https://www.tencentcloud.com/document/product/1141/69667

Commands to uninstall existing versions and install the required tilearn-llm and tilearn.ops packages, along with the colossalai dependency.

```bash
# tilearn-llm>=0.9.3
# tilearn.ops>=0.2.1.172
pip3 uninstall -y tilearn.llm tilearn.ops
pip3 install tilearn-llm==0.9.3 -i https://pypi.tuna.tsinghua.edu.cn/simple 
pip3 install tilearn.ops==0.2.1.172 -i https://g-bnvx3728-pypi.pkg.coding.net/tione/tilearn/simple
wget https://tione-public-cos-1308945662.cos.ap-shanghai.myqcloud.com/tilearn/hybrid_parallel/colossalai-0.3.4.1-cp310-cp310-linux_x86_64.whlpip3 install colossalai-0.3.4.1-cp310-cp310-linux_x86_64.whl
```

--------------------------------

### Install Python package in a specific Conda kernel

Source: https://www.tencentcloud.com/document/product/1141/69615

Use this script to install packages into a single designated Conda environment while running as the tione user.

```bash

  #!/bin/bash
  sudo -u tione -i <<'EOF'

  # This will affect only the Jupyter kernel called "conda_python3".
  source /opt/conda/bin/activate python3

  # Replace fire with the name of the package you want to install.
  pip install fire
  # You can also perform "conda install" here as well.

  source /opt/conda/bin/deactivate

  EOF

```

--------------------------------

### Define Dockerfile

Source: https://www.tencentcloud.com/document/product/1141/69622

Create a Dockerfile to package the model service and launch script into a base tiinfer image.

```dockerfile
FROM ccr.ccs.tencentyun.com/tione-public-images/ti-cloud-gpu-base-tiinfer:py38-cu111-1.0.0

COPY model_service.py /opt/model/model_service.py
COPY entrypoint.sh ./entrypoint.sh
RUN chmod +x ./entrypoint.sh
```

--------------------------------

### Modified Startup Command for TI-ONE

Source: https://www.tencentcloud.com/document/product/1141/69646

Updated command to point the model path to the platform-mounted /data/model directory.

```bash
python3 -m vllm.entrypoints.openai.api_server --model /data/model
```

--------------------------------

### Install Python package in all Conda kernels

Source: https://www.tencentcloud.com/document/product/1141/69615

Iterate through all available Conda environments to install a package globally, excluding the Jupyter system environment to maintain stability.

```bash

  #!/bin/bash
  sudo -u tione -i <<'EOF'

  # Note that "base" is special environment name, include it there as well.

  for env in base /opt/conda/envs/*; do
      source /opt/conda/bin/activate $(basename "$env")

      # Installing packages in the Jupyter system environment can affect stability of your tione
      # Notebook Instance.  You can remove this check if you'd like to install Jupyter extensions, etc.
      if [ $env = 'JupyterSystemEnv' ]; then
        continue
      fi

      # Replace myPackage with the name of the package you want to install.
      pip install fire
      # You can also perform "conda install" here as well.

      source /opt/conda/bin/deactivate
  done

  EOF

```

--------------------------------

### describe_training_tasks

Source: https://www.tencentcloud.com/document/product/1141/69714

Get a list of training tasks.

```APIDOC
## describe_training_tasks

### Description
Get training task list.

### Parameters
- **filters** (list) - Optional
- **tag_filters** (list) - Optional
- **offset** (int) - Optional
- **limit** (int) - Optional
- **order** (str) - Optional
- **order_field** (str) - Optional
```

--------------------------------

### Non-streaming API Response

Source: https://www.tencentcloud.com/document/product/1141/69646

Example of a JSON response received from a non-streaming API call.

```json
{"id":"cmpl-f2bec3ca2ded4b518fb8e73dc3461202","object":"text_completion","created":1719890717,"model":"Qwen2-7B-Instruct","choices":[{"index":0,"text":"，I've been feeling very anxious recently. What methods can I use to ease it? Hello! Anxiety is a common emotional response, but it can be mitigated by some methods. You can try deep breathing, meditation, exercise, listening to music, chatting with friends and other methods to relax yourself. Meanwhile","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":1,"total_tokens":51,"completion_tokens":50}}
```

--------------------------------

### Initialize Ray and Run Distributed Tasks

Source: https://www.tencentcloud.com/document/product/1141/69605

Use ray.init() for default cluster connection and define remote actors to perform distributed counting.

```python
import ray

ray.init()

# Define the Actor class.
@ray.remote
class Counter:
    def __init__(self):
        self.value = 0
    
    def increment(self):
        self.value += 1
        return self.value

# Create an Actor instance.
counter = Counter.remote()

# Concurrently call the Actor method.
futures = [counter.increment.remote() for _ in range(10)]
results = ray.get(futures)  # [1, 2, 3, ..., 10]

print("Counter result:", results)
```

--------------------------------

### Retrieve Training Task Logs with Pagination

Source: https://www.tencentcloud.com/document/product/1141/69714

Demonstrates how to fetch logs for a specific pod and use the next() method to handle paginated results.

```python
now_time = datetime.datetime.now(datetime.timezone.utc)
now_time_str = now_time.isoformat()
result = client.describe_train_logs("train-51cd6bf7ec1000-37c5p5nlr01s-launcher",
                                    "2021-12-10T09:32:03.823509+00:00",
                                    now_time_str,
                                    limit=30)
print(result)
print(result.next())
print(result.next())
print(result.next())
```

--------------------------------

### Build and Test Inference Image

Source: https://www.tencentcloud.com/document/product/1141/69622

Commands to build the Docker image, run it locally for testing, and push it to the registry.

```bash
docker build . --tag ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUR_IMAGENAME
```

```bash
docker run -d --name myinfer ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUT_IMAGENAME
```

```bash
docker exec -it myinfer bash
```

```bash
curl http://127.0.0.1:8501/v1/models/m:predict -d '{"a": 1, "b": 2}'
```

```bash
docker push ccr.ccs.tencentyun.com/YOUR_NAMESPACE/YOUR_IMAGENAME
```

--------------------------------

### Check for CoT in Model Response

Source: https://www.tencentcloud.com/document/product/1141/76817

Example of checking if the model response contains reasoning content.

```python
def preprocess(data: dict, resp: dict) -> bool:
    # Check whether the model response contains reasoning_content.
    return resp['reasoning_content'] == ''
```

--------------------------------

### Streaming API Response

Source: https://www.tencentcloud.com/document/product/1141/69646

Example of the data stream format returned by the API when streaming is enabled.

```text
data: {"id":"cmpl-3a575c7fd0204234afc51e195ee06596","created":1719890729,"model":"Qwen2-7B-Instruct","choices":[{"index":0,"text":",","logprobs":null,"finish_reason":null,"stop_reason":null}]}

{"id":"cmpl-3a575c7fd0204234afc51e195ee06596","created":1719890729,"model":"Qwen2-7B-Instruct","choices":[{"index":0,"text":"I","logprobs":null,"finish_reason":null,"stop_reason":null}]}

...ignorable intermediate results here...

data: {"id":"cmpl-3a575c7fd0204234afc51e195ee06596","created":1719890729,"model":"Qwen2-7B-Instruct","choices":[{"index":0,"text":".","logprobs":null,"finish_reason":null,"stop_reason":null}]}

data: {"id":"cmpl-3a575c7fd0204234afc51e195ee06596","created":1719890729,"model":"Qwen2-7B-Instruct","choices":[{"index":0,"text":"simultaneously","logprobs":null,"finish_reason":"length","stop_reason":null}]}

data: [DONE]
```

--------------------------------

### API Call via cURL

Source: https://www.tencentcloud.com/document/product/1141/69646

Examples for performing non-streaming and streaming API calls using cURL.

```bash
# Public network access address
SERVER_URL = https://ms-gp6rjk2jj-**********.gw.ap-shanghai.ti.tencentcs.com/ms-gp6rjk2j

Non-streaming call
curl -H "content-type: application/json" ${SERVER_URL}/v1/completions -d '{"model":"Qwen2-7B-Instruct","prompt":"hello","max_tokens":50,"temperature":0}'
# Streaming call
curl -H "content-type: application/json" ${SERVER_URL}/v1/completions -d '{"model":"Qwen2-7B-Instruct","prompt":"hello","max_tokens":50,"temperature":0, "stream": true}'
```

--------------------------------

### Accessing Inference Service with OpenAI SDK

Source: https://www.tencentcloud.com/document/product/1141/69648

Demonstrates how to configure the OpenAI client for local or public network access and perform both standard and streaming chat completions.

```python
import os
from openai import OpenAI

# Example 1: Container local access address
os.environ["OPENAI_BASE_URL"] = "http://127.0.0.1:8501/v1"
# Example 2: Public network access address, which can be obtained through the service invocation page (add v1 after the address when using openai sdk)
os.environ["OPENAI_BASE_URL"] = "https://service-********.sh.tencentapigw.com:443/tione/v1"
# If the --api-keys parameter is set when the service starts or the OPENAI_API_KEYS environment variable enables authentication, OPENAI_API_KEY needs to be set to any one of the available api_keys; otherwise, it can be filled in arbitrarily.
os.environ["OPENAI_API_KEY"] = "EMPTY"

client = OpenAI()

# Non-streaming request example:
print("----- standard request -----")
completion = client.chat.completions.create(
    model="model",
    messages=[
        {
            "role": "user",
            Hello
        },
    ],
    temperature=0.7,
    top_p=1.0, 
    max_tokens=128,
)
print(completion.choices[0].message.content)


# Streaming request example:
print("----- streaming request -----")
stream = client.chat.completions.create(
    model="model",
    messages=[
        {
            "role": "user",
            Hello
        },
    ],
    temperature=0.7,
    top_p=1.0,
    max_tokens=128,
    stream=True,
)
for chunk in stream:
    if not chunk.choices or not chunk.choices[0].delta.content:
        continue
    print(chunk.choices[0].delta.content, end="")
print()
```