EasyRec

https://easyrec.readthedocs.io/en/latest/
Admin
EasyRec is a recommendation framework designed for flexible model building, component reuse, and...
Tokens:282,215
Snippets:1,318
Trust Score:8.3
Update:1 year ago
Show doc for...
Tokens:
Raw
### EasyRec Configuration: Input/Output Paths

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Example protobuf configuration snippet defining input file paths for training and evaluation, and the directory for saving models.

```protobuf
train_input_path: "dwd_avazu_ctr_deepmodel_train.csv"
eval_input_path: "dwd_avazu_ctr_deepmodel_test.csv"
model_dir: "experiments/easy_rec/"
```

--------------------------------

### Protobuf Evaluation Configuration

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Configures evaluation settings for EasyRec, such as the number of examples to use for evaluation (num_examples) and the set of metrics to compute (metrics_set). This example specifies the AUC metric.

```protobuf
eval_config {
  # 仅仅评估1000个样本，这里是为了示例速度考虑，实际使用时需要删除
  num_examples: 1000
  metrics_set: {
    # metric为auc
    auc {}
  }
}
```

--------------------------------

### Run EasyRec Training (Single GPU)

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Command to start the training process for an EasyRec model on a single GPU, specifying the pipeline configuration file.

```bash
CUDA_VISIBLE_DEVICES=0 python -m easy_rec.python.train_eval --pipeline_config_path dwd_avazu_ctr_deepmodel_local.config
```

--------------------------------

### Clone and Initialize EasyRec Repository

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Instructions to clone the EasyRec GitHub repository and run the initialization script for setting up the project.

```bash
git clone https://github.com/alibaba/EasyRec.git
cd EasyRec
bash scripts/init.sh
python setup.py install
```

--------------------------------

### OSS Configuration Example

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_pai

Example configuration block for accessing data via Alibaba Cloud OSS, specifying credentials and endpoint.

```json
[Credentials]
language=ch
endpoint = oss-cn-beijing.aliyuncs.com
accessKeyID = xxx
accessKeySecret= xxx
```

--------------------------------

### Install and Use Pre-commit Hooks

Source: https://easyrec.readthedocs.io/en/latest/develop

Instructions for installing the pre-commit tool, initializing Git hooks, and running checks for code formatting and linting. This ensures code quality and consistency across the project.

```bash
pip install -U pre-commit
pre-commit install
pre-commit run -a
```

--------------------------------

### Install EasyRec with Anaconda

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Steps to set up the EasyRec environment using Anaconda, including creating a virtual environment, installing TensorFlow and TensorFlow Probability with compatible versions.

```bash
conda create -n py36_tf12 python=3.6.8
conda activate py36_tf12
pip install tensorflow==1.12.0
pip install tensorflow_probability==0.5.0
```

--------------------------------

### EasyRec Input Data Format Example

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

An example of the expected CSV format for input data used by EasyRec, noting that headers are not required.

```text
1,10,1005,0,85f751fd,c4e18dd6,50e219e0,0e8e4642,b408d42a,09481d60,a99f214a,5deb445a, f4fffcd0,1,0,2098,32,5,238,0,56,0,5
```

--------------------------------

### Run EasyRec Training (GPU PS Training)

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Command to execute distributed training using a shell script, suitable for multi-GPU setups where PS runs on CPU and masters/workers on GPUs.

```bash
wget https://easyrec.oss-cn-beijing.aliyuncs.com/scripts/train_2gpu.sh
sh train_2gpu.sh dwd_avazu_ctr_deepmodel_local.config
```

--------------------------------

### ODPS Configuration Example

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_pai

Example configuration for connecting to Alibaba Cloud MaxCompute (ODPS). Includes project name, access credentials, and service endpoints.

```ini
project_name=easy_rec_test
access_id=xxx
access_key=xxx
end_point=http://service.odps.aliyun.com/api
# MaxCompute 服务的访问链接
tunnel_endpoint=http://dt.odps.aliyun.com
# MaxCompute Tunnel 服务的访问链接
log_view_host=http://logview.odps.aliyun.com
```

--------------------------------

### EasyRec Configuration: Data Fields

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Example protobuf configuration snippet detailing data input fields, including their names, types, and default values, used for parsing input data.

```protobuf
data_config {
  separator: ","
  input_fields: {
    input_name: "label"
    input_type: FLOAT
    default_val:""
  }
  input_fields: {
    input_name: "site_id"
    input_type: STRING
    default_val:""
  }
  input_fields: {
    input_name: "site_domain"
    input_type: STRING
    default_val:""
  }
}
```

--------------------------------

### Build and Run EasyRec Docker Image (Custom)

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Instructions to build a custom Docker image for EasyRec using provided scripts and then run it, mapping local directories.

```bash
git clone https://github.com/alibaba/EasyRec.git
cd EasyRec
bash scripts/build_docker.sh
sudo docker run -td --network host -v /local_path:/docker_path mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-<easyrec_version>
sudo docker exec -it <CONTAINER_ID> bash
```

--------------------------------

### Install AutoML Package

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/hpo_pai.md

Downloads and installs the EasyRec AutoML package using pip. This is the first step to get started with PAI-HPO.

```bash
wget http://easyrec.oss-cn-beijing.aliyuncs.com/releases/pai_automl-0.0.1rc1-py3-none-any.whl
pip install pai_automl-0.0.1rc1-py3-none-any.whl
```

--------------------------------

### Run EasyRec Export (CPU)

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Command to export a trained EasyRec model for inference, specifying the pipeline configuration and output directory. This example runs on CPU.

```bash
CUDA_VISIBLE_DEVICES='' python -m easy_rec.python.export --pipeline_config_path dwd_avazu_ctr_deepmodel_local.config --export_dir dwd_avazu_ctr_export
```

--------------------------------

### Run EasyRec with Docker (Pull Image)

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Steps to pull a pre-built EasyRec Docker image and run it, mounting local EasyRec code into the container for execution.

```bash
git clone https://github.com/alibaba/EasyRec.git
cd EasyRec
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-0.8.5
docker run -td --network host -v /local_path/EasyRec:/docker_path/EasyRec mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-0.8.5
docker exec -it <CONTAINER_ID> bash
```

--------------------------------

### Install EasyRec AutoML Package

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/hpo_emr.md

Download and install the EasyRec AutoML package using wget and pip.

```bash
wget http://easyrec.oss-cn-beijing.aliyuncs.com/releases/pai_automl-0.0.1rc1-py3-none-any.whl
pip install pai_automl-0.0.1rc1-py3-none-any.whl
```

--------------------------------

### Install HPO Tools Script

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/pai_nni_hpo.md

Shell script to download and install HPO tools. It takes optional arguments for example download location and DLC version, and handles system-specific installations for Linux, macOS, and macOS ARM. It also includes steps for environment setup like sourcing .bashrc.

```shell
wget https://automl-nni.oss-cn-beijing.aliyuncs.com/nni/hpo_tools/scripts/install_hpo_tools.sh
source install_hpo_tools.sh ./ aliyun
source ~/.bashrc
```

```shell
# If on macOS, switch zsh to bash
chsh -s /bin/bash

wget https://automl-nni.oss-cn-beijing.aliyuncs.com/nni/hpo_tools/scripts/install_hpo_tools.sh
source install_hpo_tools.sh ./ mac-dlc
source ~/.bashrc
```

```shell
# If on macOS, switch zsh to bash
chsh -s /bin/bash

wget https://automl-nni.oss-cn-beijing.aliyuncs.com/nni/hpo_tools/scripts/install_hpo_tools.sh
source install_hpo_tools.sh ./ mac-arm-dlc
source ~/.bashrc
```

--------------------------------

### Launch Docker Image and Copy Examples

Source: https://easyrec.readthedocs.io/en/latest/automl/pai_nni_hpo

Demonstrates how to launch a Docker container with a specific NNI image, mount local directories, copy example files from the container, and then execute commands within the container. This is useful for running NNI examples without local installation.

```shell
mkdir -p ./examples
cd examples
echo $(pwd)

# 挂载&获取container id
container_id=`docker run -td --network host  -v $(pwd):/HpoTools/test registry.cn-shanghai.aliyuncs.com/mybigpai/nni:cpu-latest`
echo $container_id

# get examples:cp docker examples to local
docker cp $container_id:/HpoTools/examples/search $(pwd)
# 配置案例路径
ls $(pwd)/search

# 运行镜像
docker exec -ti $container_id /bin/bash
cd /HpoTools/test/search
```

--------------------------------

### Example Metric Configuration

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_res

Illustrates how multi-objective metrics are defined and displayed in the UI, showing example values for AUC and accuracy.

```text
auc=0.5
accuracy=0.5
```

--------------------------------

### PyCharm Docker Plugin Setup

Source: https://easyrec.readthedocs.io/en/latest/pycharm_vscode_docker

Steps to install the Docker plugin in PyCharm Professional (version 2022.2.2 or later) to enable Docker integration, including connecting to Docker services.

```python
# In PyCharm:
# 1. Go to Settings/Preferences -> Plugins.
# 2. Search for and install the 'Docker' plugin.
# 3. After installation, Docker connectivity will appear in the 'Services' tab at the bottom of PyCharm.
```

--------------------------------

### Start PAI-HPO Tuning

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_pai

Launches the PAI-HPO tuning process using the EasyRec Python module. Requires configuration files for ODPS, OSS, and hyperparameters.

```python
python -m easy_rec.python.hpo.pai_hpo --odps_config ~/.odps_config.ini --oss_config ~/.ossutilconfig --bucket oss://xxx --role_arn acs:ram::xxx:role/xxx --hyperparams samples/hpo/hyperparams.json  --exp_dir easy_rec_test/experiment/model_hpo  --train_tables train_longonehot_4deepfm_20 --eval_tables test_longonehot_4deepfm_20 --config_path oss://xxx/easy_rec_test/dwd_avazu_hpo.config
```

--------------------------------

### Protobuf Training Configuration

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Specifies training parameters for EasyRec, including logging frequency (log_step_count_steps), optimizer settings (e.g., Adam with exponential decay learning rate), and distributed training options (sync_replicas). It also defines the total number of training steps.

```protobuf
train_config {
  # 每200轮打印一行log
  log_step_count_steps: 200
  # 优化器相关的参数
  optimizer_config: {
    adam_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.0001
          decay_steps: 100000
          decay_factor: 0.5
          min_learning_rate: 0.0000001
        }
      }
    }
    use_moving_average: false
  }
  # 使用SyncReplicasOptimizer进行分布式训练(同步模式)
  sync_replicas: true
  # num_steps = total_sample_num * num_epochs / batch_size / num_workers
  num_steps:1000
}
```

--------------------------------

### Get EasyRec Source Code

Source: https://easyrec.readthedocs.io/en/latest/_sources/models/user_define.md

Clones the EasyRec repository from GitHub and initializes necessary scripts for setup.

```bash
git clone https://github.com/alibaba/EasyRec.git
sh scripts/init.sh
```

--------------------------------

### Install AutoML Package

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_pai

Installs the EasyRec AutoML package using pip. This is a prerequisite for using AutoML features.

```bash
wget http://easyrec.oss-cn-beijing.aliyuncs.com/releases/pai_automl-0.0.1rc1-py3-none-any.whl
pip install pai_automl-0.0.1rc1-py3-none-any.whl
```

--------------------------------

### BlockPackage Usage Example for Contrastive Learning

Source: https://easyrec.readthedocs.io/en/latest/component/backbone

Illustrates the practical application of BlockPackage in a contrastive learning setup. It shows how to define feature groups, package reusable network components (like feature encoders), and configure output blocks for training.

```protobuf
model_config {
  model_class: "RankModel"
  feature_groups {
    group_name: "all"
    feature_names: "adgroup_id"
    feature_names: "user"
    ...
    feature_names: "pid"
    wide_deep: DEEP
  }

  backbone {
    packages {
      name: 'feature_encoder'
      blocks {
        name: "fea_dropout"
        inputs {
          feature_group_name: "all"
        }
        input_layer {
          dropout_rate: 0.5
          only_output_3d_tensor: true
        }
      }
      blocks {
        name: "encode"
        inputs {
          block_name: "fea_dropout"
        }
        layers {
          keras_layer {
            class_name: 'BSTCTR'
            bst {
              hidden_size: 128
              num_attention_heads: 4
              num_hidden_layers: 3
              intermediate_size: 128
              hidden_act: 'gelu'
              max_position_embeddings: 50
              hidden_dropout_prob: 0.1
              attention_probs_dropout_prob: 0
            }
          }
        }
        layers {
          keras_layer {
            class_name: 'Dense'
            st_params {
              fields {
                key: 'units'
                value: { number_value: 128 }
              }
              fields {
                key: 'kernel_initializer'
                value: { string_value: 'zeros' }
              }
            }
          }
        }
      }
    }
    blocks {
      name: "all"
      inputs {
        name: "all"
      }
      input_layer {
        only_output_3d_tensor: true
      }
    }
    blocks {
      name: "loss_ctr"
      merge_inputs_into_list: true
      inputs {
        package_name: 'feature_encoder'
      }
      inputs {
        package_name: 'feature_encoder'
      }
      inputs {
        package_name: 'all'
      }
      keras_layer {
        class_name: 'LOSSCTR'
        st_params{
          fields {
            key: 'cl_weight'
            value: { number_value: 1 }
          }
          fields {
            key: 'au_weight'
            value: { number_value: 0.01 }
          }
        }
      }
    }
  }
  model_params {
    l2_regularization: 1e-5
  }
  embedding_regularization: 1e-5
}

```

--------------------------------

### Install EasyRec via Anaconda

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Steps to install EasyRec using Anaconda, including creating a virtual environment, installing TensorFlow and TensorFlow Probability with compatible versions, cloning the repository, and installing the package.

```conda
conda create -n py36_tf12 python=3.6.8
conda activate py36_tf12
```

```pip
pip install tensorflow==1.12.0
pip install tensorflow_probability==0.5.0
```

```shell
git clone https://github.com/alibaba/EasyRec.git
cd EasyRec
bash scripts/init.sh
python setup.py install
```

--------------------------------

### Evaluation Configuration

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Configures the evaluation process, including the number of examples to use for evaluation and the metrics to compute. This example specifies using the AUC metric.

```config
eval_config {
  # 仅仅评估1000个样本，这里是为了示例速度考虑，实际使用时需要删除
  num_examples: 1000
  metrics_set: {
    # metric为auc
    auc {}
  }
}
```

--------------------------------

### Install and Initialize Pre-commit Hooks

Source: https://easyrec.readthedocs.io/en/latest/_sources/develop.md

Installs the pre-commit tool and initializes its hooks for the repository. This ensures code quality checks and formatting are applied automatically before each commit.

```Bash
pip install -U pre-commit
pre-commit install
```

--------------------------------

### Run EasyRec Training/Evaluation/Export (CPU)

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

General command for running training, evaluation, or export tasks on CPU by not specifying the CUDA_VISIBLE_DEVICES environment variable.

```bash
python -m easy_rec.python.train_eval --pipeline_config_path dwd_avazu_ctr_deepmodel_local.config
```

--------------------------------

### Custom Command Execution with EasyRec

Source: https://easyrec.readthedocs.io/en/latest/develop

Example of how to run custom scripts or functionalities within the EasyRec framework using the 'custom' command and specifying an entry file and extra parameters.

```bash
pai -name easy_rec_ext
  -Dcmd='custom'
  -DentryFile='easy_rec/python/tools/feature_selection.py'
  -Dextra_params='--topk 1000'
```

--------------------------------

### Protobuf Feature Configuration

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Defines feature configurations for EasyRec, including input names, feature types (e.g., IdFeature), embedding dimensions, and hash bucket sizes. This section outlines how raw features are processed and mapped for model input.

```protobuf
feature_config: {
  features: {
    input_names: "hour"
    # 特征类型
    feature_type: IdFeature
    # embedding向量的dimension
    embedding_dim: 16
    # hash_bucket大小，通过tf.strings.to_hash_bucket将hour字符串映射到0-49的Id
    hash_bucket_size: 50
  }
  features: {
    input_names: "c1"
    feature_type: IdFeature
    embedding_dim: 16
    hash_bucket_size: 10
  }
  ...
  features: {
    input_names: "site_category"
    feature_type: IdFeature
    embedding_dim: 16
    hash_bucket_size: 100
  }
  features: {
    input_names: "app_id"
    feature_type: IdFeature
    embedding_dim: 32
    hash_bucket_size: 10000
  }
  ...
  features: {
    input_names: "c15"
    feature_type: IdFeature
    embedding_dim: 16
    hash_bucket_size: 500
  }
  features: {
    input_names: "c16"
    feature_type: IdFeature
    embedding_dim: 16
    hash_bucket_size: 500
  }
  ...
  features: {
    input_names: "c20"
    feature_type: IdFeature
    embedding_dim: 16
    hash_bucket_size: 500
  }
  features: {
    input_names: "c21"
    feature_type: IdFeature
    embedding_dim: 16
    hash_bucket_size: 500
  }
}
```

--------------------------------

### Git LFS Configuration Example

Source: https://easyrec.readthedocs.io/en/latest/_sources/develop.md

An example configuration file for git-lfs, specifying settings for interacting with OSS (Object Storage Service). It defines bucket names, directories, hosts, and cache locations.

```YAML
bucket_name = easyrec
git_oss_data_dir = data/git_oss_sample_data
host = oss-cn-beijing.aliyuncs.com
git_oss_cache_dir = ${TMPDIR}/${PROJECT_NAME}/.git_oss_cache
git_oss_private_config = ~/.git_oss_config_private
```

--------------------------------

### OSS Configuration Example

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/hpo_pai.md

Example JSON structure for the OSS configuration file, specifying credentials and endpoint details for accessing Object Storage Service.

```json
{
  "Credentials": {
    "language": "ch",
    "endpoint": "oss-cn-beijing.aliyuncs.com",
    "accessKeyID": "xxx",
    "accessKeySecret": "xxx"
  }
}
```

--------------------------------

### Configuration File: Input/Output Paths

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Example configuration snippet defining paths for training and evaluation data files, and the directory where the model will be saved.

```text
# 训练文件和测试文件
train_input_path: "dwd_avazu_ctr_deepmodel_train.csv"
eval_input_path: "dwd_avazu_ctr_deepmodel_test.csv"
# 模型保存路径
model_dir: "experiments/easy_rec/"
```

--------------------------------

### Build and Run Custom Docker Image

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Instructions to clone the EasyRec repository, build a Docker image using provided scripts (defaulting to TensorFlow 1.15), and run the custom image with volume mounting.

```shell
git clone https://github.com/alibaba/EasyRec.git
cd EasyRec
bash scripts/build_docker.sh
sudo docker run -td --network host -v /local_path:/docker_path mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-<easyrec_version>
sudo docker exec -it <CONTAINER_ID> bash
```

--------------------------------

### Upload Data to OSS for DLC Tutorial

Source: https://easyrec.readthedocs.io/en/latest/quick_start/dlc_tutorial

Instructions for uploading training data and configuration files to Aliyun OSS for use with EasyRec in a DLC environment. This step is crucial before creating datasets and running DLC tasks.

```bash
Upload EasyRec training data and configuration files to Aliyun OSS.

Example data: http://easyrec.oss-cn-beijing.aliyuncs.com/demo/dwd_avazu_ctr_deepmodel_10w.csv
Example config: http://easyrec.oss-cn-beijing.aliyuncs.com/demo/wide_and_deep_on_avazau_ctr.config
```

--------------------------------

### Pull and Run Pre-built Docker Image

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Steps to clone the EasyRec repository, pull a pre-built Docker image, run it in detached mode with host networking and volume mounting, and access the container.

```shell
git clone https://github.com/alibaba/EasyRec.git
cd EasyRec
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-0.8.5
docker run -td --network host -v /local_path/EasyRec:/docker_path/EasyRec mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-0.8.5
docker exec -it <CONTAINER_ID> bash
```

--------------------------------

### EasyRec Use Cases and Resources

Source: https://easyrec.readthedocs.io/en/latest/quick_start/designer_tutorial

Links to external resources for advanced EasyRec usage, including feature engineering, model training, and vector recall on Alibaba Cloud.

```APIDOC
External Resources:

- Alibaba Cloud Official Documentation for Designer Use Cases: [https://help.aliyun.com/zh/pai/use-cases/overview-18](https://help.aliyun.com/zh/pai/use-cases/overview-18)
  Covers scenarios like feature engineering, ranking model training, and vector recall.
```

--------------------------------

### Install HPO Tools Script

Source: https://easyrec.readthedocs.io/en/latest/automl/pai_nni_hpo

Downloads and sources a shell script to install HPO tools. This script handles the installation of DLC and ODPS command-line tools, registers PAIAssessor for early stopping, and downloads example configurations.

```shell
wget https://automl-nni.oss-cn-beijing.aliyuncs.com/nni/hpo_tools/scripts/install_hpo_tools.sh
source install_hpo_tools.sh ./ mac-arm-dlc

source ~/.bashrc
```

--------------------------------

### Configuration File: Data Fields

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Example configuration snippet for data processing, specifying the CSV separator and defining input fields with their names, types, and default values.

```text
# 数据相关的描述
data_config {
  # 字段之间的分隔符
  separator: ","
  # 和csv或者odps table里面字段一一对应
  input_fields: {
    input_name: "label"
    input_type: FLOAT
    default_val:""
  }
  ...
  input_fields: {
    input_name: "site_id"
    input_type: STRING
    default_val:""
  }
  input_fields: {
    input_name: "site_domain"
    input_type: STRING
    default_val:""
  }
}
```

--------------------------------

### SQL: Create and Upload Data

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/mc_tutorial_inner.md

Demonstrates creating a MaxCompute table and uploading data from a CSV file using the `tunnel upload` command. This is a prerequisite for training and evaluation.

```SQL
create table if not exists dwd_avazu_ctr_deepmodel_train (
    click bigint,
    hour string,
    c1 string,
    banner_pos string,
    site_id string,
    site_domain string,
    site_category string,
    app_id string,
    app_domain string,
    app_category string,
    device_id string,
    device_ip string,
    device_model string,
    device_type string,
    device_conn_type string,
    c14 string,
    c15 string,
    c16 string,
    c17 string,
    c18 string,
    c19 string,
    c20 string,
    c21 string
);

tunnel upload dwd_avazu_ctr_deepmodel_train.csv your_own_project.dwd_avazu_ctr_deepmodel_train;
```

--------------------------------

### VSCode Local Container Connection

Source: https://easyrec.readthedocs.io/en/latest/pycharm_vscode_docker

Guide for connecting VSCode to a running local Docker container using the 'Remote - Containers' extension. It details plugin installation and the connection process.

```python
# VSCode Extensions:
# - Remote - Containers
# - Remote - WSL
# - Remote Development

# After installation, use the Remote Explorer in VSCode to connect to running containers.
```

--------------------------------

### HardNegativeSamplerV2 Get Samples (APIDOC)

Source: https://easyrec.readthedocs.io/en/latest/api/easy_rec.python.core

The get method performs the actual sampling of negative and hard negative examples. It takes user and item IDs as input and returns a dictionary of sampled features.

```APIDOC
get(*src_ids*, *dst_ids*)
  Sampling method.
  Parameters:
    src_ids: User ID tensor.
    dst_ids: Item ID tensor.
  Returns:
    Sampled feature dict. The first batch_size elements are negative samples, and the remainder are hard negative samples.
```

--------------------------------

### CSV Data Example

Source: https://easyrec.readthedocs.io/en/latest/automl/auto_cross_emr

Illustrates the expected CSV format for input data, with columns separated by commas. This format is used for training and testing datasets.

```text
1,10,1005,0,85f751fd,c4e18dd6,50e219e0,0e8e4642,b408d42a,09481d60,a99f214a,5deb445a, f4fffcd0,1,0,2098,32,5,238,0,56,0,5
```

--------------------------------

### Training Configuration

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Specifies parameters for the training process, including logging frequency, optimizer settings (like Adam with exponential decay), and distributed training options. It also defines the total number of training steps.

```config
# 训练相关的参数
train_config {
  # 每200轮打印一行log
  log_step_count_steps: 200
  # 优化器相关的参数
  optimizer_config: {
    adam_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.0001
          decay_steps: 100000
          decay_factor: 0.5
          min_learning_rate: 0.0000001
        }
      }
    }
    use_moving_average: false
  }
  # 使用SyncReplicasOptimizer进行分布式训练(同步模式)
  sync_replicas: true
  # num_steps = total_sample_num * num_epochs / batch_size / num_workers
  num_steps:1000
}
```

--------------------------------

### Start Finetune Tuning

Source: https://easyrec.readthedocs.io/en/latest/automl/finetune_config

Command to initiate hyperparameter tuning for fine-tuning tasks using nnictl, specifying the configuration file and port.

```shell
nnictl create --config config_finetune.yml --port=8617
```

--------------------------------

### Contrastive Learning Configuration (Protobuf)

Source: https://easyrec.readthedocs.io/en/latest/_sources/component/backbone.md

Configuration for a contrastive learning model using EasyRec's block package feature. It defines user and item towers, an auxiliary loss for contrastive learning, and a top MLP. This example highlights reusable sub-networks and parameter sharing for self-supervised learning scenarios.

```protobuf
model_config: {
  model_name: "ContrastiveLearning"
  model_class: "RankModel"
  feature_groups: {
    group_name: 'user'
    feature_names: 'user_id'
    feature_names: 'job_id'
    feature_names: 'age'
    feature_names: 'gender'
    wide_deep: DEEP
  }
  feature_groups: {
    group_name: 'item'
    feature_names: 'movie_id'
    feature_names: 'year'
    feature_names: 'genres'
    wide_deep: DEEP
  }
  backbone {
    blocks {
      name: 'user_tower'
      inputs {
        feature_group_name: 'user'
      }
      keras_layer {
        class_name: 'MLP'
        mlp {
          hidden_units: [256, 128]
        }
      }
    }
    packages {
      name: 'item_tower'
      blocks {
        name: 'item'
        inputs {
          feature_group_name: 'item'
        }
        input_layer {
          dropout_rate: 0.2
        }
      }
      blocks {
        name: 'item_encoder'
        inputs {
          block_name: 'item'
        }
        keras_layer {
          class_name: 'MLP'
          mlp {
            hidden_units: [256, 128]
          }
        }
      }
    }
    blocks {
      name: 'contrastive_learning'
      inputs {
        package_name: 'item_tower'
      }
      inputs {
        package_name: 'item_tower'
      }
      merge_inputs_into_list: true
      keras_layer {
        class_name: 'AuxiliaryLoss'
        st_params {
          fields {
            key: 'loss_type'
            value: { string_value: 'info_nce' }
          }
          fields {
            key: 'loss_weight'
            value: { number_value: 0.1 }
          }
          fields {
            key: 'temperature'
            value: { number_value: 0.2 }
          }
        }
      }
    }
    blocks {
      name: 'top_mlp'
      inputs {
        block_name: 'contrastive_learning'
        ignore_input: true
      }
      inputs {
        block_name: 'user_tower'
      }
      inputs {
        package_name: 'item_tower'
        reset_input {}
      }
      keras_layer {
        class_name: 'MLP'
        mlp {
          hidden_units: [128, 64]
        }
      }
    }
    concat_blocks: 'top_mlp'
  }
  model_params {
    l2_regularization: 1e-4
  }
  embedding_regularization: 1e-4
}
```

--------------------------------

### Tuning Result - Metric Information

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_pai

Example JSON objects containing performance metrics for tuning experiments, such as loss and AUC, recorded at different global steps.

```json
{"loss": 0.6656426191329956, "auc": 0.4911114573478699, "loss/loss/total_loss": 0.6656426191329956, "global_step": 0, "loss/loss/cross_entropy_loss": 0.6656426191329956}
```

```json
{"loss": 0.16333971917629242, "auc": 0.6694868803024292, "loss/loss/total_loss": 0.16333971917629242, "global_step": 1000, "loss/loss/cross_entropy_loss": 0.16333971917629242}
```

--------------------------------

### PAI TensorFlow Cluster Configuration

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_pai

Example JSON structure for defining the cluster configuration, specifying the number and resources for PS (Parameter Server) and worker nodes.

```json
--cluster='{"ps": {"count":1, "cpu":1000}, "worker" : {"count":3, "cpu":1000, "gpu":100, "memory":40000}}'
```

--------------------------------

### EasyRec Configuration: Evaluation Metrics

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/mc_tutorial_inner.md

Configures evaluation parameters, such as the number of examples to use for evaluation (`num_examples`) and the set of metrics to compute (e.g., AUC). The `num_examples` is often reduced for demonstration purposes.

```protobuf
eval_config {
  # 仅仅评估1000个样本，这里是为了示例速度考虑，实际使用时需要删除
  num_examples: 1000
  metrics_set: {
    # metric为auc
    auc {}
  }
}
```

--------------------------------

### GPU PS Training Script

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Executes a shell script to manage distributed training with a GPU Parameter Server (PS) setup. Assumes PS runs on CPU, master on GPU:0, and worker on GPU:1.

```shell
wget https://easyrec.oss-cn-beijing.aliyuncs.com/scripts/train_2gpu.sh
sh train_2gpu.sh dwd_avazu_ctr_deepmodel_local.config
```

--------------------------------

### Run EasyRec Evaluation (Single GPU)

Source: https://easyrec.readthedocs.io/en/latest/_sources/quick_start/local_tutorial.md

Command to perform model evaluation using EasyRec on a single GPU, referencing the pipeline configuration file.

```bash
CUDA_VISIBLE_DEVICES=0 python -m easy_rec.python.eval --pipeline_config_path dwd_avazu_ctr_deepmodel_local.config
```

--------------------------------

### Test Single Pre-check

Source: https://easyrec.readthedocs.io/en/latest/api/easy_rec.python.utils

Performs a pre-check for a single pipeline configuration. This function is useful for validating pipeline setup and initial conditions before starting training.

```python
easy_rec.python.utils.test_utils.test_single_pre_check(*pipeline_config_path*, *test_dir*)

Test single pre-check.
```

--------------------------------

### Sequence Component Configuration Example

Source: https://easyrec.readthedocs.io/en/latest/component/sequence

Demonstrates the configuration for sequence models, including feature groups and backbone blocks for input layers and DIN attention. It shows how to define feature names and structure the model's input and processing pipeline.

```protobuf
model_config: {
  model_name: 'DIN'
  model_class: 'RankModel'
  ...
  feature_groups: {
    group_name: 'sequence'
    feature_names: "cate_id"
    feature_names: "brand"
    feature_names: "tag_category_list"
    feature_names: "tag_brand_list"
    wide_deep: DEEP
  }
  backbone {
    blocks {
      name: 'seq_input'
      inputs {
        feature_group_name: 'sequence'
      }
      input_layer {
        output_seq_and_normal_feature: true
      }
    }
    blocks {
      name: 'DIN'
      inputs {
        block_name: 'seq_input'
      }
      keras_layer {
        class_name: 'DIN'
        din {
          attention_dnn {
            hidden_units: 32
            hidden_units: 1
            activation: "dice"
          }
          need_target_feature: true
        }
      }
    }
    ...
  }
}

```

--------------------------------

### Tuning Result - Parameter Information

Source: https://easyrec.readthedocs.io/en/latest/automl/hpo_pai

Example JSON object showing the parameters used for a specific tuning experiment run, including the hyperparameter names and their assigned values.

```json
{"id": 0, "experiment": "default", "run": "tuning_experiment", "param": {"feature_config.features[input_names[0]=field1].embedding_dim;feature_config.features[input_names[0]=field20].embedding_dim": "32;32"}, "checkpoint_id": -1, "custom": {}}
```

--------------------------------

### Hyperparameter Configuration - Integer

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/hpo_emr.md

Example JSON structure for defining an integer hyperparameter search space. Includes the parameter name, type, minimum value, and maximum value.

```json
{
  "type": "Integer",
  "name": "weeks",
  "min_value": 0,
  "max_value": 5
}
```

--------------------------------

### NNI HPO Configuration: trial.ini (MaxCompute)

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/pai_nni_hpo.md

Example configuration file for a trial run on the PAI MaxCompute platform. It includes OSS and ODPS configurations, parameter mapping, and the PAI command for training, specifying model versions, data tables, cluster resources, and model directories.

```ini
[oss_config]
endpoint=http://oss-cn-beijing.aliyuncs.com
accessKeyID=${AK}
accessKeySecret=${SK}

[odps_config]
access_id=${AK}
access_key=${SK}
project_name=pai_rec_dev
end_point=http://service.odps.aliyun.com/api

[params_config]
# easy_rec config replace KEY->search params value
params_src_dst_filepath=/root/test_nni/AutoML/HpoTools/hpo_tools/search/maxcompute_easyrec/samples/pipeline_params.config,oss://lcl-bj/eval_test/config/easyrec_model_${exp_id}_${trial_id}.config

[platform_config]name=MaxCompute
cmd1=PAI -name=easy_rec_ext
            -project=algo_public
            -Dversion="0.4.2"
            -Dconfig='oss://lcl-bj/eval_test/config/easyrec_model_${exp_id}_${trial_id}.config'
            -Dcmd='train'
            -Dtrain_tables='odps://pai_rec_dev/tables/dwd_avazu_ctr_deepmodel_train_10000'
            -Deval_tables='odps://pai_rec_dev/tables/dwd_avazu_ctr_deepmodel_test_1000'
            -Dcluster='{"ps":{"count":1,"cpu":1000 } },"worker":{"count":3,"cpu":1000,"memory":40000}}'
            -Darn='xxx'
            -Dbuckets='oss://lcl-bj/'
            -Dmodel_dir='oss://lcl-bj/eval_dist_test/model_${exp_id}_${trial_id}'
            -DossHost='oss-cn-beijing-internal.aliyuncs.com'
            -Deval_method='separate'

[metric_config]
# metric type is summary/table
metric_type=summary
metric_source=oss://lcl-bj/eval_dist_test/model_${exp_id}_${trial_id}/eval_val/
# best/final/avg,default=best
final_mode=final
metric_dict={'auc':1}
```

--------------------------------

### Hyperparameter Configuration - Real

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/hpo_emr.md

Example JSON structure for defining a real (float) hyperparameter search space. Specifies the parameter name, type, minimum value, and maximum value.

```json
{
  "type": "Real",
  "name": "epsilon",
  "min_value": 0.0001,
  "max_value": 0.1
}
```

--------------------------------

### Simple Data Configuration Example

Source: https://easyrec.readthedocs.io/en/latest/feature/data

Demonstrates a basic data configuration for EasyRec, specifying input fields like 'click', 'uid', and 'item_id'. It uses OdpsInputV2 for MaxCompute or suggests CSVInput for local training, along with batch size and epoch settings.

```config
data_config {
  batch_size: 2048
  input_fields {
    input_name: "click"
    input_type: INT32
  }
  input_fields {
    input_name: "uid"
    input_type: STRING
  }
  input_fields {
    input_name: "item_id"
    input_type: STRING
  }
  label_fields: "click"
  num_epochs: 1
  prefetch_size: 32
  input_type: OdpsInputV2
}
```

--------------------------------

### Get Base Ports for Workers

Source: https://easyrec.readthedocs.io/en/latest/api/easy_rec.python.utils

Calculates and returns base port numbers suitable for distributed worker setups. The number of workers is a key input for determining the port allocation.

```python
def get_ports_base(num_worker):
```

--------------------------------

### GPU Single-Node Single-GPU Training

Source: https://easyrec.readthedocs.io/en/latest/quick_start/local_tutorial

Command to initiate training on a single GPU, specifying the pipeline configuration file. It uses the `train_eval` module.

```shell
CUDA_VISIBLE_DEVICES=0 python -m easy_rec.python.train_eval --pipeline_config_path dwd_avazu_ctr_deepmodel_local.config
```

--------------------------------

### Setup GraphLearn Import and String Attributes

Source: https://easyrec.readthedocs.io/en/latest/_modules/easy_rec/python/core/sampler

Configures the string_attrs function for GraphLearn and includes error handling for GraphLearn installation. It maps internal data types to GraphLearn string representations.

```python
def string_attrs(self, string_attrs):
  self._string_attrs = self._reshape(string_attrs, expand_shape=True)
  self._inited = True

try:
  import graphlearn as gl
  from graphlearn.python.data.values import Values
  Values.string_attrs = string_attrs
except Exception:
  logging.info(
      'GraphLearn is not installed. You can install it by "pip install https://easyrec.oss-cn-beijing.aliyuncs.com/3rdparty/graphlearn-0.7-cp27-cp27mu-linux_x86_64.whl"'  # noqa: E501
  )
```

--------------------------------

### EasyRec Model Serving: Command-line Deployment

Source: https://easyrec.readthedocs.io/en/latest/index

Explains how to deploy EasyRec models for online serving using command-line tools. This is a common method for production environments.

```APIDOC
model_serving_cli_deployment:
  # Deploying EasyRec models for real-time inference via command line.
  # This often involves starting a server process that loads the exported model.
```

--------------------------------

### Run a Specific Unit Test Case

Source: https://easyrec.readthedocs.io/en/latest/develop

Example of how to run a single, specific unit test case using Python's unittest module, targeting a particular test class and method.

```python
TEST_DEVICES='' python -m easy_rec.python.test.train_eval_test TrainEvalTest.test_tfrecord_input
```

--------------------------------

### Hyperparameter Configuration - Categorical

Source: https://easyrec.readthedocs.io/en/latest/_sources/automl/hpo_emr.md

Example JSON structure for defining a categorical hyperparameter search space. Specifies the parameter name (full path in EasyRec config), type, and a list of candidate values.

```json
[
  {
    "type": "Categorical",
    "name": "feature_config.features[input_names[0]=field1].embedding_dim",
    "candidates": ["16", "32", "48", "64", "80"]
  }
]
```

--------------------------------

### Create Sample View for Feature Joining

Source: https://easyrec.readthedocs.io/en/latest/feature/odl_sample

Creates a temporary view `sample_view` by joining `odl_sample_with_lbl` and `odl_callback_log`. It filters callback logs for 'item' module, selects the latest feature per request/item, and joins with sample data based on request ID, item ID, and a time window.

```SQL
create temporary view sample_view as
     select a.request_id, a.user_id, a.item_id, a.ln_play_time, a.is_valid_play, feature, b.request_time
     from  odl_sample_with_lbl a
     inner join (
          select * from (
           select request_id, item_id, request_time, generate_features as feature, ts,
             row_number() over(partition by request_id, item_id order by proctime() asc) as rn
           from odl_callback_log
           where `module` = 'item' and (generate_features is not null and generate_features <> '')
          ) where rn = 1
     ) b
     on a.request_id = b.request_id and a.item_id = b.item_id
     where a.ts between b.ts - INTERVAL '30' SECONDS  and b.ts + INTERVAL '30' MINUTE;
```

--------------------------------

### TensorBoard Logging for Model Evaluation

Source: https://easyrec.readthedocs.io/en/latest/optimize

Instructions on how to use TensorBoard to visualize model performance on the evaluation set at different checkpoints. It requires downloading event files and running the tensorboard command, pointing to the directory containing these files.

```bash
tensorboard --logdir=events_file_path
```
Tokens:
Raw