Maintainers
Pricing
Enterprise
More
More
Documentation
Try Live
Rankings
Install
Blog
Docs7
Ask
New
Add Docs
PaddlePaddle Serving
https://paddlepaddle.github.io/Serving/
Admin
PaddlePaddle distributed training and Serving streamlined deployment.
Tokens:
8,227
Snippets:
100
Trust Score:
9.7
Update:
3 months ago
Context
Chat
Benchmark
Latest
Show doc for...
Code
Info
Show Results
Tokens:
Raw
Copy
Link
Tokens:
Raw
Copy
Link
### Install Go Environment Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Commands to download, extract, and configure the Go 1.12 environment. ```bash wget https://studygolang.com/dl/golang/go1.12.7.linux-amd64.tar.gz --no-check-certificate ``` ```bash tar zxvf go1.12.7.linux-amd64.tar.gz -C /path/to/go ``` ```bash export GOROOT=/path/to/go export PATH=/path/to/go/bin:$PATH ``` -------------------------------- ### Start cube-transfer service Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Launch the service using the specified port, log level, and configuration file path. ```bash ./cube-transfer -p 8099 -l 4 --config conf/transfer.conf ``` -------------------------------- ### Configure Cube Server Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Example configuration for the cube server, including port settings and memory mode. ```text --port=8000 --dict_split=1 --in_mem=true ``` -------------------------------- ### Start Cube Server and Agent Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Commands to launch the cube server and the associated cube-agent with a specified port. ```bash nohup bin/cube & nohup bin/cube-agent -P 8001 & ``` -------------------------------- ### Install Volcano Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Deploy the Volcano batch scheduling system to the Kubernetes cluster. ```bash $ kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml ``` -------------------------------- ### Install and Manage Volcano Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Commands to install the Volcano framework and handle potential installation failures. ```bash kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml ``` ```bash kubectl delete -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml ``` -------------------------------- ### GET /instance/status Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Retrieves the current status of the cube-transfer instance. ```APIDOC ## GET /instance/status ### Description Retrieves the current status of the cube-transfer instance. ### Method GET ### Endpoint /instance/status ``` -------------------------------- ### GET /dict/info Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Retrieves the current status of the dictionary. ```APIDOC ## GET /dict/info ### Description Retrieves the current status of the dictionary managed by the cube-transfer service. ### Method GET ### Endpoint /dict/info ``` -------------------------------- ### Example Plaintext Input Format Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The required format for plaintext input files, where each line contains a key-value pair separated by a colon. ```text 1:1 2:2 10:10 11 12 11:this is eleven 12:value can string 1676869128226002114:48241 37064 91 -539 114 51 -122 269 229 -134 -282 1657749292782759014:167 40 98 27 117 10 -29 15 74 67 -54 ``` -------------------------------- ### GET /dict/deploy/history Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Retrieves the deployment history from the most recent base to the currently deploying delta. ```APIDOC ## GET /dict/deploy/history ### Description Retrieves the deployment history from the most recent base to the currently deploying delta. ### Method GET ### Endpoint /dict/deploy/history ``` -------------------------------- ### 更新模型文件 Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html 下载并替换Serving目录下的模型文件，随后重启服务。 ```bash $ cd data/model/paddle/fluid $ wget http://${FILE_SERVER_IP}:${FILE_SERVER_PORT}/data/ctr_model.tar.gz # `FILE_SERVER_IP`与`FILE_SERVER_PORT`请参考1.4节获取。 $ tar zxvf ctr_model.tar.gz # 假设解压出一个inference_only目录 $ rm -rf ctr_prediction/* # 删除旧的ctr_prediction目录下内容 $ cp inference_only/* ctr_prediction/ $ cd ../../../../ # 切换至serving所在目录 $ ls bin conf data kvdb log $ killall serving # 杀死旧的serving进程 $ bin/serving & # 重启serving ``` -------------------------------- ### View Cube-Builder Help and Flags Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Display the available command-line flags and configuration options for the cube-builder tool. ```bash cube-builder: Usage : ./cube-builder --help Flags from /home/work/cube-builder/src/main.cpp: -cur_version (current version, no need) type: int32 default: 0 //单机builder模式下不需要 -depend_version (depend version, job mode delta need) type: int32 default: 0 //单机builder base模式下不需要，patch模式找到meta_info里的base的key -dict_name (dict name, no need) type: string default: "" //词典名，单机builder模式下不加默认空，用来和版本拼接生成索引文件名 -input_path (source data input path) type: string default: "" //必须，源数据所在路径，仅支持本地路径 -job_mode (job mode base/delta default:base) type: string default: "base" //默认base模式，delta模式：-job_mode=delta -last_version (last version, job mode delta need) type: int32 default: 0 //单机builder base模式下不需要，patch模式找到meta_info里的base的id -master_address (master address, no need) type: string default: "" //单机builder模式不需要，会把index meta信息写到本地output/meta_info目录下 -only_build (wheather build need transfer) type: bool default: true //单机builder模式不需要，代表是不是单机builder，如果false会向master_address发送请求，将index meta信息写到远程 -output_path (source data input path) type: string default: "" //必须，索引建库数据的输出路径，仅支持本地路径 -shard_num (shard num) type: int32 default: -1 //必须，数据切分的分片数量 ``` -------------------------------- ### Deploy Components via Script Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Execute the automated deployment script for all components. ```bash $ bash paddle-suite.sh ``` -------------------------------- ### Execute Training Tasks Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Commands to set up permissions, label nodes, and launch the CTR training job. ```bash kubectl create -f defaultserviceaccountclusterrole.yaml ``` ```bash kubectl label node ${NODE NAME} nodeType=model ``` ```bash kubectl apply -f volcano-ctr-demo-baiduyun.yaml ``` -------------------------------- ### 配置model_toolkit.prototxt Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html 定义Serving引擎配置，包括模型路径及CTR预估相关的远程参数服务设置。 ```protobuf engines { name: "image_classification_resnet" type: "FLUID_CPU_NATIVE_DIR" reloadable_meta: "./data/model/paddle/fluid_time_file" reloadable_type: "timestamp_ne" model_data_path: "./data/model/paddle/fluid/SE_ResNeXt50_32x4d" runtime_thread_num: 0 batch_infer_size: 0 enable_batch_align: 0 } engines { name: "text_classification_bow" type: "FLUID_CPU_ANALYSIS_DIR" reloadable_meta: "./data/model/paddle/fluid_time_file" reloadable_type: "timestamp_ne" model_data_path: "./data/model/paddle/fluid/text_classification_lstm" runtime_thread_num: 0 batch_infer_size: 0 enable_batch_align: 0 } engines { name: "ctr_prediction" type: "FLUID_CPU_ANALYSIS_DIR" reloadable_meta: "./data/model/paddle/fluid_time_file" reloadable_type: "timestamp_ne" model_data_path: "./data/model/paddle/fluid/ctr_prediction" runtime_thread_num: 0 batch_infer_size: 0 enable_batch_align: 0 sparse_param_service_type: REMOTE sparse_param_service_table_name: "test_dict" } ``` -------------------------------- ### Manage File Server Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Commands to deploy and verify the file server. ```bash $ kubectl apply -f fileserver.yaml ``` ```bash $ kubectl get pod ``` ```bash $ kubectl get service ``` -------------------------------- ### Manage Cube Parameter Server Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Commands to deploy and verify the Cube sparse parameter server. ```bash $ kubectl apply -f cube.yaml ``` ```bash $ kubectl get service ``` -------------------------------- ### Execute Client Binary Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Run the compiled client binary to initiate the prediction service, with logs generated in the ./log directory. ```bash ./bin/ctr_prediction ``` -------------------------------- ### Configure kv_to_seqfile Paths Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Set the base donefile and source file paths within the kv_to_seqfile.py script. ```python BASE_DONEFILE = DATA_PATH + "donefile/base.txt" #base donefile文件地址 SOURCE_FILE = './source/file.txt' #明文源数据路径 ``` -------------------------------- ### Compile Cube Components Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Standard compilation steps for Paddle Serving including Cube components, requiring a Go compiler. ```bash $ git clone https://github.com/PaddlePaddle/Serving.git $ cd Serving $ mkdir build $ cd build $ cmake -DWITH_GPU=OFF .. # 不需要GPU $ make -jN # 这里可修改并发编译线程数 $ make install $ cd output/ $ ls bin cube cube-builder cube-transfer pdcodegen $ ls conf gflags.conf transfer.conf ``` -------------------------------- ### Run Cube-Builder in Base Mode Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Execute the builder in base mode using absolute paths for input and output. ```bash ./cube-builder -input_path=${source} -output_path=${output} -shard_num=2 -dict_name=test ``` -------------------------------- ### Verify cube-transfer file structure Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The directory structure should include the executable, configuration, and necessary working directories. ```bash $ tree . |-- cube-transfer |-- output |-- tmp `-- conf |-- transfer.conf ``` -------------------------------- ### Manage Paddle Serving Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Commands to deploy and verify the Paddle Serving service. ```bash $ kubectl apply -f paddleserving.yaml ``` ```bash $ kubectl get pod ``` ```bash $ kubectl get service ``` -------------------------------- ### 编译Paddle Serving Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html 从源码克隆并编译Paddle Serving，生成可执行文件及配置文件。 ```bash $ git clone https://github.com/PaddlePaddle/Serving.git $ cd Serving $ mkdir build $ cd build $ cmake -DWITH_GPU=OFF .. # 不需要GPU $ make -jN # 这里可修改并发编译线程数 $ make install $ cd output/demo/serving $ ls bin conf data kvdb ``` -------------------------------- ### Cube-Builder Directory Structure Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The expected directory layout for the cube-builder tool. ```bash $ tree `-- cube-builder |-- source | `-- test_seqfile |-- output ``` -------------------------------- ### Configure kubectl for Kubernetes Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Move the downloaded cluster configuration file to the default kubectl path. ```bash $ mv kubectl.conf ~/.kube/config ``` -------------------------------- ### Manage Cube Transfer Tool Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Commands to deploy, verify, and monitor the Cube transfer tool. ```bash $ kubectl apply -f transfer.yaml ``` ```bash $ kubectl get pod ``` ```bash $ kubectl logs cube-transfer ``` -------------------------------- ### Cube File Structure Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Expected file structure after copying binaries and configuration files to physical nodes. ```text $ tree . |-- bin | `-- cube | `-- cube-agent |-- conf | `-- gflags.conf ``` -------------------------------- ### Execute Redis Performance Test Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Run the Redis client benchmark tool with specified host, thread count, request count, and batch size. ```bash $ ./get_values -h 192.168.1.1 -t 3 -r 10000 -b 1000 ``` -------------------------------- ### Base Mode Output Directory Structure Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The resulting directory structure after running a base mode build. ```bash $ tree `-- cube-builder |-- source | `-- test_seqfile `-- output |-- 1565323045_1565323045 | |-- test_part0 | | |-- data.0 | | |-- data.n | | |-- index.0 | | `-- index.n | |-- test_part0.tar | |-- test_part0.tar.md5 | |-- test_part1 | | |-- data.0 | | |-- data.n | | |-- index.0 | | `-- index.n | |-- test_part1.tar | `-- test_part1.tar.md5 `-- meta_info |-- 1565323045_1565323045_0_0.json `-- 1565323045_1565323045_1_0.json ``` -------------------------------- ### 修改gflags配置 Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html 启用Cube服务支持。 ```bash --enable_cube=true ``` -------------------------------- ### Run Cube-Builder in Delta Mode Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Execute the builder in delta mode, requiring dependency on previous version IDs. ```bash ./cube-builder -input_path=${source} -output_path=${output} -shard_num=2 -depend_version=1565323045 -last_version=1565323045 -job_mode=delta -dict_name=test ``` -------------------------------- ### HTTP File Server Configuration Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html YAML configuration for deploying a static file server and a load balancer service to expose model files. ```yaml apiVersion: v1 kind: Pod metadata: name: file-server labels: app: file-server spec: nodeSelector: nodeType: model volumes: - hostPath: path: /home/work type: "" name: file-home containers: - name: file-server image: halverneus/static-file-server ports: - containerPort: 8080 volumeMounts: - mountPath: /web name: file-home --- kind: Service apiVersion: v1 metadata: name: loadbalancer spec: type: LoadBalancer ports: - name: file-server port: 8080 targetPort: 8080 selector: app: file-server ``` -------------------------------- ### 下载模型参数 Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html 通过HTTP服务下载CTR预估模型压缩包。 ```bash wget http://${FILE_SERVER_IP}:${FILE_SERVER_PORT}/data/ctr_model.tar.gz ``` -------------------------------- ### Configure RBAC for Training Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html YAML configuration to create a ClusterRole and ClusterRoleBinding for the default service account. ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: default namespace: default rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: default namespace: default subjects: - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: default apiGroup: rbac.authorization.k8s.io ``` -------------------------------- ### Execute Distributed Training Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Commands to deploy and monitor the CTR distributed training job. ```bash $ kubectl apply -f ctr.yaml ``` ```bash $ kubectl get pod ``` ```bash $ kubectl logs edl-demo-trainer-0 ``` -------------------------------- ### Read Seqfile with kvtool Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Use the kvtool.py script to read and output the contents of a seqfile. ```python python kvtool.py /home/work/test/seqfile ``` -------------------------------- ### Configure cube-transfer Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The configuration file conf/transfer.conf requires absolute paths and specific network settings for the transfer service and cube agents. ```ini [default] dict_name: test_dict # 词典名 mode: base_delta # 配送模式base_only/base_delta download_mode: http # 配送方式，可以选择http或ftp wget_port: 80 # http服务的端口 buildtool_local: /path/to/cube-builder # builder工具位置，必须在本地，绝对路径 donefile_address: http://${FILE_SERVER_IP}:${FILE_SERVER_PORT}/data/ctr_cube/donefile/ # donefile路径，${FILE_SERVER_IP}:${FILE_SERVER_PORT}为1.4节搭建的file server地址。文件夹内包含base.txt, patch.txt和一批Hadoop SequenceFile文件 output_address: /some/path/to/output # builder产出的数据索引输出位置 tmp_address: /some/path/to/tmp # transfer工具运行中临时文件存放位置 shard_num: 2 # 分片数 copy_num: 1 # 每片副本数 deploy_path: /home/work/test_dict # 不用修改 transfer_address: 10.10.10.5 # cube-transfer本机的ip [cube_agent] agent0_0: 192.168.1.1:8001 # 0号分片0号副本的agent ip:port cube0_0: 192.168.1.1:8000:/path/to/cube # 0号分片0号副本的cube，该路径下会存放配送的数据，需要与启动cube可执行文件时所在的路径一致 格式：ip:port:deploy_path agent1_0: 192.168.1.2:8001 # 1号分片0号副本的agent ip:port cube1_0: 192.168.1.2:8000:/path/to/cube # 1号分片0号副本的cube，该路径下会存放配送的数据，需要与启动cube可执行文件时所在的路径一致 格式：ip:port:deploy_path ``` -------------------------------- ### CTR预估模型配置项 Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html CTR预估模型中关于远程参数服务的关键配置。 ```protobuf sparse_param_service_type: REMOTE sparse_param_service_table_name: "test_dict" ``` -------------------------------- ### Deploy and Execute CTR Prediction Client Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Commands to apply the client configuration and execute the prediction task within the container. ```bash $ kubectl apply -f paddleclient.yaml ``` ```bash # 进入容器 $ kubectl exec -ti pdservingclient /bin/bash # 此命令在容器内执行 $ bin/ctr_prediction ``` -------------------------------- ### Update and Push Docker Images Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Methods for updating the environment by committing changes to a container or rebuilding from a Dockerfile. ```bash $ docker commit ${DOCKER_CONTAINER_NAME} ${DOCKER_IMAGE_NAME} $ docker push ${DOCKER_IMAGE_NAME} ``` ```bash $ docker build -t ${DOCKER_IMAGE_NAME} . $ docker push ${DOCKER_IMAGE_NAME} ``` -------------------------------- ### Delta Mode Output Directory Structure Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The resulting directory structure after running a delta mode build. ```bash $ tree `-- cube-builder |-- source | `-- test_seqfile `-- output |-- 1565323045_1565323045 | |-- test_part0 | | |-- data.0 | | |-- data.n | | |-- index.0 | | `-- index.n | |-- test_part0.tar | |-- test_part0.tar.md5 | |-- test_part1 | | |-- data.0 | | |-- data.n | | |-- index.0 | | `-- index.n | |-- test_part1.tar | `-- test_part1.tar.md5 |-- 1565323045_1565326078 | |-- test_part0 | |-- data.0 | | |-- data.n | | |-- index.0 | | `-- index.n | |-- test_part0.tar | |-- test_part0.tar.md5 | |-- test_part1 | | |-- data.0 | | |-- data.n | | |-- index.0 | | `-- index.n | |-- test_part1.tar | `-- test_part1.tar.md5 `-- meta_info |-- 1565323045_1565323045_0_0.json |-- 1565323045_1565323045_0_0.json |-- 1565326078_1565323045_0_0.json `-- 1565326078_1565323045_1_0.json ``` -------------------------------- ### Configure Helm TLS Environment Variables Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Set these environment variables on the development machine to enable secure communication with the Tiller service. ```bash export HELM_TLS_ENABLE=true export HELM_TLS_CA_CERT=/etc/kubernetes/pki/ca.pem export HELM_TLS_CERT=/etc/kubernetes/pki/admin.pem export HELM_TLS_KEY=/etc/kubernetes/pki/admin-key.pem ``` -------------------------------- ### Configure CTR Prediction Service Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html Modify the predictors.prototxt file to define the service name, endpoint router, and cluster naming configuration. ```protobuf predictors { name: "ctr_prediction_service" service_name: "baidu.paddle_serving.predictor.ctr_prediction.CTRPredictionService" endpoint_router: "WeightedRandomRender" weighted_random_render_conf { variant_weight_list: "50" } variants { tag: "var1" naming_conf { cluster: "list://127.0.0.1:8010" } } } ``` -------------------------------- ### 配置Cube客户端 Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html 定义用于Cube客户端API的配置文件，包含分片、超时及节点列表信息。 ```json [{ "dict_name": "test_dict", "shard": 2, "dup": 1, "timeout": 200, "retry": 3, "backup_request": 100, "type": "ipport_list", "load_balancer": "rr", "nodes": [{ "ipport_list": "list://192.168.1.1:8000" },{ "ipport_list": "list://192.168.1.2:8000" }] }] ``` -------------------------------- ### Set Kubernetes Role Binding Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Grant necessary permissions for pod-to-pod communication. ```bash $ kubectl create rolebinding default-view --clusterrole=view --serviceaccount=default:default --namespace=default ``` -------------------------------- ### Access Sparse Parameter Donefile Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The URL path for the donefile used to describe the sparse parameter directory structure. ```text http://${FILE_SERVER_IP}:${FILE_SERVER_PORT}/data/ctr_cube/donefile/base.txt ``` -------------------------------- ### Define donefile format Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The donefile uses JSON format to track base and patch data updates for the cube-transfer service. ```json {"id":"1562000400","key":"1562000400","input":"/home/work/test_data/input/seqfile"} ``` ```json {"id":"1562000401","key":"1562000400","input":"/home/work/test_data/input/seqfile"} {"id":"1562000402","key":"1562000400","input":"/home/work/test_data/input/seqfile"} ``` -------------------------------- ### Access Dense Model Archive Source: https://paddlepaddle.github.io/Serving/doc/DEPLOY.html The URL path for the dense model tarball hosted on the HTTP file server. ```text http://${FILE_SERVER_IP}:${FILE_SERVER_PORT}/data/ctr_model.tar.gz ``` -------------------------------- ### Label Kubernetes Node Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Assign a specific label to a node for model output and file services. ```bash $ kubectl label nodes $NODE_NAME nodeType=model ``` -------------------------------- ### Verify Kubernetes Node Status Source: https://paddlepaddle.github.io/Serving/doc/ELASTIC_CTR.html Check the status of nodes in the cluster. ```bash $ kubectl get node ```
