### Qlib Config YAML Example

Source: https://rdagent.readthedocs.io/en/latest/scens/model_agent_fin.html

Example configuration snippet from config.yaml for running a developed model in Qlib. Specifies the market setting.

```yaml
market : Specifies the market, which is set to csi300.
```

--------------------------------

### Install Azure CLI

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Download and install the Azure CLI using curl.

```sh
curl -L https://aka.ms/InstallAzureCli | bash
```

--------------------------------

### Start RDAgent Web App

Source: https://rdagent.readthedocs.io/en/latest/ui.html

Run this command in the RD-Agent/ folder to start the web application. Specify the port and log directory. The --debug flag is optional and enables additional features.

```bash
rdagent ui --port <port> --log-dir <log_dir like "log/"> [--debug]
```

--------------------------------

### RD-Agent Environment Setup

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Initializes the data preparation process for a specified competition. This script should be run from the 'source_data' directory.

```python
if __name__ == "__main__":
    competitions = "playground-series-s4e9"
    raw = Path(__file__).resolve().parent
    prepare(
        raw=raw,
        public=raw.parent.parent / competitions,
        private=raw.parent.parent / "eval" / competitions,
    )
```

--------------------------------

### Download Customized Dataset

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Use this command to download a pre-prepared dataset for reference. Ensure you have wget installed.

```bash
wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip
```

--------------------------------

### Competition Description File Example

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Example content for a description.md file, including competition name, overview, evaluation metrics (RMSE), submission file format, and timeline. This file should be placed in the competition's data directory.

```markdown
# Competition name: playground-series-s4e9

## Overview

**Welcome to the 2024 Kaggle Playground Series!** We plan to continue in the spirit of previous playgrounds, providing interesting and approachable datasets for our community to practice their machine learning skills, and anticipate a competition each month.

**Your Goal:** The goal of this competition is to predict the price of used cars based on various attributes.

## Evaluation

### Root Mean Squared Error (RMSE)

Submissions are scored on the root mean squared error. RMSE is defined as:

$$
\mathrm{RMSE} = \left( \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2 \right)^{\frac{1}{2}}
$$

where $\hat{y}_i$ is the predicted value and $y_i$ is the original value for each instance $i$.

### Submission File

For each `id` in the test set, you must predict the `price` of the car. The file should contain a header and have the following format:

```
id,price
188533,43878.016
188534,43878.016
188535,43878.016
etc.
```

## Timeline
- **Start Date** - September 1, 2024
- **Entry Deadline** - Same as the Final Submission Deadline
- **Team Merger Deadline** - Same as the Final Submission Deadline
- **Final Submission Deadline** - September 30, 2024

All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.

## About the Tabular Playground Series

The goal of the Tabular Playground Series is to provide the Kaggle community with a variety of fairly light-weight challenges that can be used to learn and sharpen skills in different aspects of machine learning and data science. The duration of each competition will generally only last a few weeks, and may have longer or shorter durations depending on the challenge. The challenges will generally use fairly light-weight datasets that are synthetically generated from real-world data, and will provide an opportunity to quickly iterate through various model and feature engineering ideas, create visualizations, etc.

### Synthetically-Generated Datasets

Using synthetic data for Playground competitions allows us to strike a balance between having real-world data (with named features) and ensuring test labels are not publicly available. This allows us to host competitions with more interesting datasets than in the past. While there are still challenges with synthetic data generation, the state-of-the-art is much better now than when we started the Tabular Playground Series two years ago, and that goal is to produce datasets that have far fewer artifacts. Please feel free to give us feedback on the datasets for the different competitions so that we can continue to improve!

## Prizes
- 1st Place - Choice of Kaggle merchandise
- 2nd Place - Choice of Kaggle merchandise
- 3rd Place - Choice of Kaggle merchandise

**Please note**: In order to encourage more participation from beginners, Kaggle merchandise will only be awarded once per person in this series. If a person has previously won, we'll skip to the next team.

## Citation


```

--------------------------------

### Download Competition Data

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Use this command to download a specific competition dataset. Ensure you have `wget` installed.

```bash
wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/playground-series-s4e9.zip
```

--------------------------------

### Run RD Agent Benchmark

Source: https://rdagent.readthedocs.io/en/latest/research/benchmark.html

Execute the benchmark script after completing installation and configuration. A .pkl file will be generated upon completion.

```bash
dotenv run -- python rdagent/app/benchmark/factor/eval.py

```

--------------------------------

### Example MLE-Bench Data Structure

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

This illustrates the expected folder structure for competition data after RD-Agent has downloaded and processed it for MLE-Bench.

```tree
ds_data
├── tabular-playground-series-dec-2021
│   ├── description.md
│   ├── sample_submission.csv
│   ├── test.csv
│   └── train.csv
└── zip_files
    └── tabular-playground-series-dec-2021
        └── tabular-playground-series-dec-2021.zip

```

--------------------------------

### Configure DeepSeek Chat and SiliconFlow Embedding

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

This example sets up DeepSeek for chat models and SiliconFlow for embedding models. Ensure you replace placeholders with your actual API keys and note the 'litellm_proxy' prefix for the embedding model.

```shell
# CHAT MODEL: Using DeepSeek Official API
CHAT_MODEL=deepseek/deepseek-chat
DEEPSEEK_API_KEY=<replace_with_your_deepseek_api_key>

# EMBEDDING MODEL: Using SiliconFlow for embedding since DeepSeek has no embedding model.
# Note: embedding requires litellm_proxy prefix
EMBEDDING_MODEL=litellm_proxy/BAAI/bge-m3
LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

```

--------------------------------

### Run RDAgent Finance Application

Source: https://rdagent.readthedocs.io/en/latest/scens/data_agent_fin.html

Executes the RDAgent application for finance factor analysis. This command starts the agent after installation and environment setup.

```bash
rdagent fin_factor
```

--------------------------------

### Install RDAgent Package

Source: https://rdagent.readthedocs.io/en/latest/scens/model_copilot_general.html

Installs the RDAgent package from PyPI using pip. This command should be run within the activated conda environment.

```bash
pip install rdagent
```

--------------------------------

### Download and Unzip Reports

Source: https://rdagent.readthedocs.io/en/latest/scens/data_copilot_fin.html

Downloads a zip archive of financial reports using wget and then extracts them into the 'git_ignore_folder/reports' directory. Ensure wget is installed.

```bash
wget https://github.com/SunsetWolf/rdagent_resource/releases/download/reports/all_reports.zip
unzip all_reports.zip -d git_ignore_folder/reports
```

--------------------------------

### Configure Fin-Model Scenario Time Segments

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Set custom start and end dates for training, validation, and testing segments in the fin_model scenario. Defaults are provided if not specified.

```bash
QLIB_MODEL_TRAIN_START=<train start date, default is 2008-01-01>
QLIB_MODEL_TRAIN_END=<train end date, default is 2014-12-31>
QLIB_MODEL_VALID_START=<valid start date, default is 2015-01-01>
QLIB_MODEL_VALID_END=<valid end date, default is 2016-12-31>
QLIB_MODEL_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_MODEL_TEST_END=<test / backtest end date, default is 2020-12-31>
```

--------------------------------

### Load Sparse Features with Python

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Use the 'sparse' package to load .npz files containing time-dependent features. Ensure the package is installed.

```python
import sparse

X = sparse.load_npz("<url>/X.npz").todense()
```

--------------------------------

### Configure Fin-Factor Scenario Time Segments

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Set custom start and end dates for training, validation, and testing segments in the fin_factor scenario. Defaults are provided if not specified.

```bash
QLIB_FACTOR_TRAIN_START=<train start date, default is 2008-01-01>
QLIB_FACTOR_TRAIN_END=<train end date, default is 2014-12-31>
QLIB_FACTOR_VALID_START=<valid start date, default is 2015-01-01>
QLIB_FACTOR_VALID_END=<valid end date, default is 2016-12-31>
QLIB_FACTOR_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_FACTOR_TEST_END=<test / backtest end date, default is 2020-12-31>
```

--------------------------------

### Set Environment Variables for RD-Agent

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Configure the RD-Agent environment by setting local data paths and the scenario class. Ensure RD-Agent is installed and configured before use.

```bash
dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
```

```bash
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
```

--------------------------------

### Run Finance Model Agent

Source: https://rdagent.readthedocs.io/en/latest/scens/model_agent_fin.html

Executes the Finance Model Agent application. This is the primary command to start the automated trading and model evolution process.

```bash
rdagent fin_model
```

--------------------------------

### Prepare Development Environment

Source: https://rdagent.readthedocs.io/en/latest/development.html

Set up the necessary development environment for RD-Agent using the provided Makefile command.

```bash
make dev
```

--------------------------------

### Clone RD-Agent Repository

Source: https://rdagent.readthedocs.io/en/latest/development.html

Clone the RD-Agent repository from GitHub to start development or contribute.

```bash
git clone https://github.com/microsoft/RD-Agent
```

--------------------------------

### CheckpointSelector Class

Source: https://rdagent.readthedocs.io/en/latest/api_reference.html

Abstract class for selecting a checkpoint within a trace to start a new experiment.

```APIDOC
## class _rdagent.core.proposal.CheckpointSelector

### Description
In the trace, we may start from any check point (we’ll represent it as a variable from_checkpoint_idx).

### Abstract Methods
#### _get_selection

##### Description
Checkpoint index represents the place where we want to create a new node. The return value should be the index of the target node (the parent of the new generating node).
- (-1, ) represents starting from the latest trial in the trace.
- (idx, ) represents starting from the idx-th trial in the trace.
- None represents starting from scratch (start a new trace).

##### Parameters
- **_trace** (Trace_) - The trace object.
```

--------------------------------

### Run Finance Quant Application

Source: https://rdagent.readthedocs.io/en/latest/scens/quant_agent_fin.html

Execute the RD-Agent finance quant application using this command. This command initiates the framework for quantitative finance tasks.

```bash
rdagent fin_quant
```

--------------------------------

### Log in to Azure

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Log in to your Azure environment using the device code flow.

```sh
az login --use-device-code
```

--------------------------------

### ExpPlanner Class

Source: https://rdagent.readthedocs.io/en/latest/api_reference.html

Abstract class for planning experiments based on the current trace.

```APIDOC
## class _rdagent.core.proposal.ExpPlanner

### Description
An abstract class for planning the experiment. The planner should generate a plan for the experiment based on the trace.

### Parameters
- **_scen** (Scenario_) - The scenario for planning.

### Abstract Methods
#### _plan

##### Description
Generate a plan for the experiment based on the trace. The plan should be a dictionary that contains the plan to each stage.

##### Parameters
- **_trace** (Trace_) - The trace object.
```

--------------------------------

### Create Conda Environment

Source: https://rdagent.readthedocs.io/en/latest/scens/model_copilot_general.html

Creates a new conda environment named 'rdagent' with Python 3.10. This is a prerequisite for installing and running RDAgent.

```bash
conda create -n rdagent python=3.10
```

--------------------------------

### Create Data Directory and Set Environment Variable

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Create a directory for storing data science data and set the KG_LOCAL_DATA_PATH environment variable using the dotenv command. Replace <your local directory> with the desired path.

```bash
mkdir -p <your local directory>/ds_data
dotenv set KG_LOCAL_DATA_PATH <your local directory>/ds_data

```

--------------------------------

### Show RD Agent Benchmark Results

Source: https://rdagent.readthedocs.io/en/latest/research/benchmark.html

Analyze the benchmark results by running the analysis script with the path to the generated .pkl file. This converts the data into a PNG image.

```bash
dotenv run -- python rdagent/app/benchmark/factor/analysis.py <log/path to.pkl>

```

--------------------------------

### Prepare Customized Datasets Script

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Python script to split raw data into training, testing, and submission files. Requires numpy, pandas, and sparse libraries. Sets up directory structure for competition data.

```python
import random
from pathlib import Path

import numpy as np
import pandas as pd
import sparse

CURRENT_DIR = Path(__file__).resolve().parent
ROOT_DIR = CURRENT_DIR.parent.parent

raw_feature_path = CURRENT_DIR / "X.npz"
raw_label_path = CURRENT_DIR / "ARF_12h.csv"

public = ROOT_DIR / "arf-12-hours-prediction-task"
private = ROOT_DIR / "eval" / "arf-12-hours-prediction-task"

if not (public / "test").exists():
    (public / "test").mkdir(parents=True, exist_ok=True)

if not (public / "train").exists():
    (public / "train").mkdir(parents=True, exist_ok=True)

if not private.exists():
    private.mkdir(parents=True, exist_ok=True)

SEED = 42
random.seed(SEED)
np.random.seed(SEED)

X_sparse = sparse.load_npz(raw_feature_path)  # COO matrix, shape: [N, D, T]
df_label = pd.read_csv(raw_label_path)  # Contains column 'ARF_LABEL'
N = X_sparse.shape[0]

indices = np.arange(N)
np.random.shuffle(indices)
split = int(0.7 * N)
train_idx, test_idx = indices[:split], indices[split:]

X_train = X_sparse[train_idx]
X_test = X_sparse[test_idx]

df_train = df_label.iloc[train_idx].reset_index(drop=True)
df_test = df_label.iloc[test_idx].reset_index(drop=True)

submission_df = df_test.copy()
submission_df["ARF_LABEL"] = 0
submission_df.drop(submission_df.columns.difference(["ID", "ARF_LABEL"]), axis=1, inplace=True)
submission_df.to_csv(public / "sample_submission.csv", index=False)

df_test.to_csv(private / "submission_test.csv", index=False)

df_test.drop(["ARF_LABEL"], axis=1, inplace=True)
df_test.to_csv(public / "test" / "ARF_12h.csv", index=False)
sparse.save_npz(public / "test" / "X.npz", X_test)

sparse.save_npz(public / "train" / "X.npz", X_train)
df_train.to_csv(public / "train" / "ARF_12h.csv", index=False)

assert (
    X_train.shape[0] == df_train.shape[0]
), f"Mismatch: X_train rows ({X_train.shape[0]}) != df_train rows ({df_train.shape[0]})")
assert (
    X_test.shape[0] == df_test.shape[0]
), f"Mismatch: X_test rows ({X_test.shape[0]}) != df_test rows ({df_test.shape[0]})")
assert df_test.shape[1] == 2, "Public test set should have 2 columns"
assert df_train.shape[1] == 3, "Public train set should have 3 columns"
assert len(df_train) + len(df_test) == len(
    df_label
), "Length of new_train and new_test should equal length of old_train"

```

--------------------------------

### Activate Conda Environment

Source: https://rdagent.readthedocs.io/en/latest/scens/model_copilot_general.html

Activates the previously created 'rdagent' conda environment. Ensure this environment is active before proceeding with RDAgent installation and execution.

```bash
conda activate rdagent
```

--------------------------------

### Prepare Customized Datasets

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

This function samples data and copies additional files to create a customized dataset. It handles both file copying and directory tree copying.

```python
def sample_and_copy_subfolder(input_dir: Path, output_dir: Path, min_frac: float = 0.02, min_num: int = 10, seed: int = 42):
    df_sample.to_csv(output_dir / "ARF_12h.csv", index=False)

    print(f"[INFO] Sampled {n_keep} of {N} from {input_dir.name}")

    # Copy additional files
    for f in input_dir.glob("*"):
        if f.name not in {"X.npz", "ARF_12h.csv"} and f.is_file():
            shutil.copy(f, output_dir / f.name)
            print(f"[COPY] Extra file: {f.name}")
```

```python
def copy_other_file(source: Path, target: Path):
    for item in source.iterdir():
        if item.name in {"train", "test"}:
            continue

        relative_path = item.relative_to(source)
        target_path = target / relative_path

        if item.is_dir():
            shutil.copytree(item, target_path, dirs_exist_ok=True)
            print(f"[COPY DIR] {item} -> {target_path}")
        elif item.is_file():
            target_path.parent.mkdir(parents=True, exist_ok=True)
            shutil.copy2(item, target_path)
            print(f"[COPY FILE] {item} -> {target_path}")
```

```python
def create_debug_data(
    dataset_path: str,
    output_path: str,
    min_frac: float = 0.02,
    min_num: int = 10,
):
    dataset_root = Path(dataset_path) / "arf-12-hours-prediction-task"
    output_root = Path(output_path)

    for sub in ["train", "test"]:
        input_dir = dataset_root / sub
        output_dir = output_root / sub
        print(f"\n[PROCESS] {sub} subset")
        sample_and_copy_subfolder(
            input_dir=input_dir,
            output_dir=output_dir,
            min_frac=min_frac,
            min_num=min_num,
            seed=42 if sub == "train" else 123,
        )
    print(dataset_root.resolve())
    print(output_root.resolve())
    copy_other_file(source=dataset_root, target=output_root)

    print(f"\n[INFO] Sampling complete → Output in: {output_root}")


if __name__ == "__main__" or globals().get("__name__") == "<run_path>":
    dataset_path = globals().get("dataset_path", "./")
    output_path = globals().get("output_path", "./sample")
    create_debug_data(
        dataset_path=dataset_path,
        output_path=output_path,
        min_frac=0.02,
        min_num=10,
    )
```

--------------------------------

### Run Kaggle Competition

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Execute a Kaggle competition task. This command is recommended for Kaggle scenarios.

```bash
rdagent kaggle --competition playground-series-s4e8
```

--------------------------------

### Adjust Benchmark Test Rounds via Environment Variable

Source: https://rdagent.readthedocs.io/en/latest/research/benchmark.html

Example of how to modify the `bench_test_round` configuration from its default value of 10 to 2 by setting an environment variable.

```bash
BENCHMARK_BENCH_TEST_ROUND=2

```

--------------------------------

### Visualize R&D Process

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Launch a web UI to visualize the R&D process logs. Specify the port and the directory containing the logs. Set `data_science` to True for data science specific visualizations.

```bash
rdagent ui --port <custom port> --log-dir <your log folder like "log/"> --data_science True
```

--------------------------------

### Run FT-Agent for LLM Fine-Tuning

Source: https://rdagent.readthedocs.io/en/latest/scens/finetune.html

Execute the FT-Agent scenario for LLM fine-tuning. Ensure required FT_* settings are configured before running. This command initiates the fine-tuning process with a specified base model.

```bash
rdagent llm_finetune --base-model Qwen/Qwen2.5-7B-Instruct

```

--------------------------------

### ExpGen Class

Source: https://rdagent.readthedocs.io/en/latest/api_reference.html

Abstract class for generating experiments based on the trace.

```APIDOC
## class _rdagent.core.proposal.ExpGen

### Description
Abstract class for generating experiments based on the trace.

### Abstract Methods
#### _gen

##### Description
Generate the experiment based on the trace. Planning is part of gen, but since we may support multi-stage planning, we need to pass plan as optional argument.
ExpGen().gen() play a role like

##### Parameters
- **_trace** (Trace_) - The trace object.
```

--------------------------------

### ExpGen.reset()

Source: https://rdagent.readthedocs.io/en/latest/api_reference.html

Resets the proposal to the initial state. Sometimes the main loop may want to reset the whole process to the initial state. Default implementation does nothing; override in subclasses if needed.

```APIDOC
## ExpGen.reset()

### Description
Resets the proposal to the initial state. Sometimes the main loop may want to reset the whole process to the initial state. Default implementation does nothing; override in subclasses if needed.

### Method
ExpGen.reset()

### Parameters
None

### Request Example
None

### Response
None
```

--------------------------------

### Download Kaggle Competition Data

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Use this command to download the raw data for a specific Kaggle competition, such as 'playground-series-s4e9'.

```bash
kaggle competitions download -c playground-series-s4e9

```

--------------------------------

### Prepare Kaggle Competition Data (Python)

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

This Python script preprocesses raw Kaggle competition data, splitting it into training and testing sets, and generating submission files. It requires pandas and scikit-learn.

```python
from pathlib import Path

import pandas as pd
from sklearn.model_selection import train_test_split


def prepare(raw: Path, public: Path, private: Path):

    # Create train and test splits from train set
    old_train = pd.read_csv(raw / "train.csv")
    new_train, new_test = train_test_split(old_train, test_size=0.1, random_state=0)

    # Create sample submission
    sample_submission = new_test.copy()
    sample_submission["price"] = 43878.016
    sample_submission.drop(sample_submission.columns.difference(["id", "price"]), axis=1, inplace=True)
    sample_submission.to_csv(public / "sample_submission.csv", index=False)

    # Create private files
    new_test.to_csv(private / "submission_test.csv", index=False)

    # Create public files visible to agents

```

--------------------------------

### Enable MLE-Bench Data Download

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Set the DS_IF_USING_MLE_DATA environment variable to True to enable automatic dataset downloads for MLE-Bench supported competitions.

```bash
dotenv set DS_IF_USING_MLE_DATA True

```

--------------------------------

### Run Linting and Checking

Source: https://rdagent.readthedocs.io/en/latest/development.html

Execute linting and checking tools to ensure code quality and adherence to standards.

```bash
make lint
```

--------------------------------

### Run Debug Data Creation Script

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Executes a Python script for creating debug data with custom sampling parameters. Requires specifying dataset path, competition name, sampling ratio (min_frac), and minimum number of samples (min_num). Ensure DS_SAMPLE_DATA_BY_LLM is set to False.

```bash
python rdagent/app/data_science/debug.py --dataset_path <dataset path> --competition <competiton_name> --min_frac <sampling ratio> --min_num <minimum number of sampling>
dotenv set DS_SAMPLE_DATA_BY_LLM False
```

--------------------------------

### Configure Unified API Base for LiteLLM

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Use this configuration when a single API base serves both chat and embedding models. Ensure the API key and base URL are correctly set.

```shell
# Set to any model supported by LiteLLM.
CHAT_MODEL=gpt-4o
EMBEDDING_MODEL=text-embedding-3-small
# Configure unified API base
# The backend api_key fully follows the convention of litellm.
OPENAI_API_BASE=<your_unified_api_base>
OPENAI_API_KEY=<replace_with_your_openai_api_key>

```

--------------------------------

### Calculate Liquidity Imbalance (Version 1)

Source: https://rdagent.readthedocs.io/en/latest/_static/RD2bench.json

Calculates the liquidity imbalance using minute trading data. This version uses the formula (bid_size-ask_size)/(bid_size+ask_size) for imbalance. Requires pandas and HDF5 files.

```python
import pandas as pd
data_hf = pd.read_hdf('high_freq.h5')
sample_df= data_hf.reset_index()
# Convert 'datetime' column to datetime and extract date for grouping
sample_df['date'] = sample_df['datetime'].dt.date
sample_df['liquidity_imbalance']=(sample_df['bidV']-sample_df['askV'])/(sample_df['bidV']+sample_df['askV'])
# Group by instrument and date
grouped = sample_df.groupby(['date','instrument'])['liquidity_imbalance']
# Calculate mean and standard deviation of the volume for each group
stats = grouped.agg(['mean', 'std'])
# Calculate Z value for each instrument per day
stats['liquidity_imbalance'] = stats['std'] / stats['mean']
# Display the calculated Z values
result=stats['liquidity_imbalance']
result.index.names = ['datetime','instrument']
# result = result.swaplevel().sort_index()
result.to_hdf('result.h5', key='data')
```

--------------------------------

### Set Environment Variables for Data Science Scenarios

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Configure essential environment variables for data science scenarios. Set the scenario class, local data path, and enable coding on the whole pipeline.

```bash
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
dotenv set DS_CODER_ON_WHOLE_PIPELINE True
```

--------------------------------

### Hypothesis2Experiment.convert()

Source: https://rdagent.readthedocs.io/en/latest/api_reference.html

Connects the hypothesis proposal to its concrete implementation.

```APIDOC
## Hypothesis2Experiment.convert()

### Description
Connects the hypothesis proposal to its concrete implementation.

### Method
Hypothesis2Experiment.convert(_hypothesis : Hypothesis_, _trace : Trace_)

### Parameters
* **_hypothesis** (Hypothesis_) - The hypothesis object.
* **_trace** (Trace_) - The trace object.

### Request Example
None

### Response
None
```

--------------------------------

### Configure MLE-Bench Competition Scenario

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Set the DS_SCEN environment variable to rdagent.scenarios.data_science.scen.KaggleScen to specify the Kaggle competition scenario for MLE-Bench.

```bash
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen

```

--------------------------------

### Enable Reasoning Thought Process Output

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Set this environment variable to 'True' if your reasoning models include a thought process in their responses and you wish to capture it.

```shell
REASONING_THINK_RM=True

```

--------------------------------

### Enable LLM Data Sampling for MLE-Bench

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Set the DS_SAMPLE_DATA_BY_LLM environment variable to True to allow the LLM to automatically extract a minimum dataset for MLE-Bench competitions.

```bash
dotenv set DS_SAMPLE_DATA_BY_LLM True

```

--------------------------------

### Summarize Scores with MLE-Bench

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Summarize scores for competitions supported by MLE-Bench or when full output is configured. Provide the parent directory of the log folder.

```bash
rdagent grade_summary --log-folder=<url_to_log>
```

--------------------------------

### Calculate Alpha PV Difference (10 Days)

Source: https://rdagent.readthedocs.io/en/latest/_static/RD2bench.json

Calculates alpha_pv_diff using 10-day changes in close and open prices. Useful for volume and price analysis.

```python
import pandas as pd
data_pv = pd.read_hdf('daily_pv.h5')
new_df= data_pv.reset_index()
# Calculate Alpha101
new_df['result'] = (new_df['$close'].diff(10) - new_df['$open'].diff(10)) / (new_df['$high'] - new_df['$low'] + 0.001)
# keep the index of the original dataframe
result=pd.DataFrame(new_df['result']).set_index(data_pv.index)
# transfer the result to series
result=result['result']
result.to_hdf('result.h5', key='data')
```

--------------------------------

### Calculate Alpha PV Difference (20 Days)

Source: https://rdagent.readthedocs.io/en/latest/_static/RD2bench.json

Calculates alpha_pv_diff using 20-day changes in close and open prices. Useful for longer-term price-volume analysis.

```python
import pandas as pd
data_pv = pd.read_hdf('daily_pv.h5')
new_df= data_pv.reset_index()
# Calculate Alpha101
new_df['result'] = (new_df['$close'].diff(20) - new_df['$open'].diff(20)) / (new_df['$high'] - new_df['$low'] + 0.001)
# keep the index of the original dataframe
result=pd.DataFrame(new_df['result']).set_index(data_pv.index)
# transfer the result to series
result=result['result']
result.to_hdf('result.h5', key='data')
```

--------------------------------

### Run General Model Copilot

Source: https://rdagent.readthedocs.io/en/latest/scens/model_copilot_general.html

Executes the General Model Copilot scenario. Replace '<path_to_pdf_file>' with the actual path to the PDF report file containing model information. This command must be run within the activated 'rdagent' conda environment.

```bash
rdagent general_model --report-file-path=<path_to_pdf_file>
```

--------------------------------

### Calculate Liquidity Imbalance (Version 2)

Source: https://rdagent.readthedocs.io/en/latest/_static/RD2bench.json

Calculates the liquidity imbalance using minute trading data. This version uses the formula (bid_size-ask_size)/2*(bid_size+ask_size) for imbalance. Requires pandas and HDF5 files.

```python
import pandas as pd
data_hf = pd.read_hdf('high_freq.h5')
sample_df= data_hf.reset_index()
# Convert 'datetime' column to datetime and extract date for grouping
sample_df['date'] = sample_df['datetime'].dt.date
sample_df['liquidity_imbalance']=(sample_df['bidV']-sample_df['askV'])/((sample_df['bidV']+sample_df['askV'])*2)
# Group by instrument and date
grouped = sample_df.groupby(['date','instrument'])['liquidity_imbalance']
# Calculate mean and standard deviation of the volume for each group
stats = grouped.agg(['mean', 'std'])
# Calculate Z value for each instrument per day
stats['liquidity_imbalance'] = stats['std'] / stats['mean']
# Display the calculated Z values
result=stats['liquidity_imbalance']
result.index.names = ['datetime','instrument']
# result = result.swaplevel().sort_index()
result.to_hdf('result.h5', key='data')
```

--------------------------------

### Configure Fin-Quant Scenario Time Segments

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Set environment variables to define training, validation, and testing time segments for factor, model, and quant stages in the fin_quant scenario. These variables are for UI display and do not affect execution.

```env
QLIB_FACTOR_TRAIN_START=<train start date, default is 2008-01-01>
QLIB_FACTOR_TRAIN_END=<train end date, default is 2014-12-31>
QLIB_FACTOR_VALID_START=<valid start date, default is 2015-01-01>
QLIB_FACTOR_VALID_END=<valid end date, default is 2016-12-31>
QLIB_FACTOR_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_FACTOR_TEST_END=<test / backtest end date, default is 2020-12-31>

QLIB_MODEL_TRAIN_START=<train start date, default is 2008-01-01>
QLIB_MODEL_TRAIN_END=<train end date, default is 2014-12-31>
QLIB_MODEL_VALID_START=<valid start date, default is 2015-01-01>
QLIB_MODEL_VALID_END=<valid end date, default is 2016-12-31>
QLIB_MODEL_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_MODEL_TEST_END=<test / backtest end date, default is 2020-12-31>

QLIB_QUANT_TRAIN_START=<train start date, default is 2008-01-01>
QLIB_QUANT_TRAIN_END=<train end date, default is 2014-12-31>
QLIB_QUANT_VALID_START=<valid start date, default is 2015-01-01>
QLIB_QUANT_VALID_END=<valid end date, default is 2016-12-31>
QLIB_QUANT_TEST_START=<test / backtest start date, default is 2017-01-01>
QLIB_QUANT_TEST_END=<test / backtest end date, default is 2020-12-31>
```

--------------------------------

### Qlib Model Configuration Fields

Source: https://rdagent.readthedocs.io/en/latest/scens/model_agent_fin.html

Details the configuration fields for the Qlib model, including their types, default values, and purpose. These fields control various aspects of model execution and data handling.

```python
env_prefix : _str = QLIB_MODEL__
protected_namespaces : _tuple = ()
```

```python
_field _coder _: str_ _ = 'rdagent.scenarios.qlib.developer.model_coder.QlibModelCoSTEER'_
```

```python
_field _evolving_n _: int_ _ = 10_
```

```python
_field _hypothesis2experiment _: str_ _ = 'rdagent.scenarios.qlib.proposal.model_proposal.QlibModelHypothesis2Experiment'_
```

```python
_field _hypothesis_gen _: str_ _ = 'rdagent.scenarios.qlib.proposal.model_proposal.QlibModelHypothesisGen'_
```

```python
_field _runner _: str_ _ = 'rdagent.scenarios.qlib.developer.model_runner.QlibModelRunner'_
```

```python
_field _scen _: str_ _ = 'rdagent.scenarios.qlib.experiment.model_experiment.QlibModelScenario'_
```

```python
_field _summarizer _: str_ _ = 'rdagent.scenarios.qlib.developer.feedback.QlibModelExperiment2Feedback'_
```

```python
_field _test_end _: str | None_ _ = '2020-08-01'_
```

```python
_field _test_start _: str_ _ = '2017-01-01'_
```

```python
_field _train_end _: str_ _ = '2014-12-31'_
```

```python
_field _train_start _: str_ _ = '2008-01-01'_
```

```python
_field _valid_end _: str_ _ = '2016-12-31'_
```

```python
_field _valid_start _: str_ _ = '2015-01-01'_
```

--------------------------------

### Data Science Configuration Settings

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Python Pydantic model for configuring data science scenarios. It includes settings for Kaggle integration, scenario classes, planners, interactors, and various timeouts.

```python
from pathlib import Path
from typing import Literal

from pydantic_settings import SettingsConfigDict

from rdagent.app.kaggle.conf import KaggleBasePropSetting


class DataScienceBasePropSetting(KaggleBasePropSetting):
    # TODO: Kaggle Setting should be the subclass of DataScience
    model_config = SettingsConfigDict(env_prefix="DS_", protected_namespaces=())

    # Main components
    ## Scen
    scen: str = "rdagent.scenarios.data_science.scen.KaggleScen"
    """
    Scenario class for data science tasks.
    - For Kaggle competitions, use: "rdagent.scenarios.data_science.scen.KaggleScen"
    - For custom data science scenarios, use: "rdagent.scenarios.data_science.scen.DataScienceScen"
    """

    planner: str = "rdagent.scenarios.data_science.proposal.exp_gen.planner.DSExpPlannerHandCraft"
    hypothesis_gen: str = "rdagent.scenarios.data_science.proposal.exp_gen.router.ParallelMultiTraceExpGen"
    interactor: str = "rdagent.components.interactor.SkipInteractor"
    trace_scheduler: str = "rdagent.scenarios.data_science.proposal.exp_gen.trace_scheduler.RoundRobinScheduler"
    """Hypothesis generation class"""

    summarizer: str = "rdagent.scenarios.data_science.dev.feedback.DSExperiment2Feedback"
    summarizer_init_kwargs: dict = {
        "version": "exp_feedback",
    }
    ## Workflow Related
    consecutive_errors: int = 5

    ## Coding Related
    coding_fail_reanalyze_threshold: int = 3

    debug_recommend_timeout: int = 600
    """The recommend time limit for running on debugging data"""
    debug_timeout: int = 600
    """The timeout limit for running on debugging data"""
    full_recommend_timeout: int = 3600
    """The recommend time limit for running on full data"""
    full_timeout: int = 3600
    """The timeout limit for running on full data"""

    #### model dump
    enable_model_dump: bool = False
    enable_doc_dev: bool = False
    model_dump_check_level: Literal["medium", "high"] = "medium"

    #### MCP documentation search integration
    enable_mcp_documentation_search: bool = False
    """Enable MCP documentation search for error resolution. Requires MCP_ENABLED=true and MCP_CONTEXT7_ENABLED=true in environment."""

    ### specific feature

    ### notebook integration
    enable_notebook_conversion: bool = False

    #### enable specification
    spec_enabled: bool = True

    #### proposal related
    # proposal_version: str = "v2" deprecated

    coder_on_whole_pipeline: bool = True
    max_trace_hist: int = 3

    coder_max_loop: int = 10
    runner_max_loop: int = 3

    sample_data_by_LLM: bool = True
    use_raw_description: bool = False
    show_nan_columns: bool = False

    ### knowledge base
    enable_knowledge_base: bool = False
    knowledge_base_version: str = "v1"
    knowledge_base_path: str | None = None
    idea_pool_json_path: str | None = None

    ### archive log folder after each loop
    enable_log_archive: bool = True
    log_archive_path: str | None = None
    log_archive_temp_path: str | None = (
        None  # This is to store the mid tar file since writing the tar file is preferred in local storage then copy to target storage
    )

    #### Evaluation on Test related
    eval_sub_dir: str = "eval"  # TODO: fixme, this is not a good name
    """We'll use f"{DS_RD_SETTING.local_data_path}/{DS_RD_SETTING.eval_sub_dir}/{competition}"
    to find the scriipt to evaluate the submission on test"""

    """---below are the settings for multi-trace---"""

    ### multi-trace related
    max_trace_num: int = 1
    """The maximum number of traces to grow before merging"""

    scheduler_temperature: float = 1.0

```

--------------------------------

### Run Data Science Competition

Source: https://rdagent.readthedocs.io/en/latest/scens/data_science.html

Execute a data science competition task. Replace `<Competition ID>` with the actual competition identifier.

```bash
rdagent data_science --competition <Competition ID>
```

```bash
rdagent data_science --competition arf-12-hours-prediction-task
```

--------------------------------

### Configure Azure OpenAI via LiteLLM with External Embeddings

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

Set CHAT_MODEL for Azure OpenAI and EMBEDDING_MODEL for an external provider like SiliconFlow. Ensure all necessary Azure OpenAI environment variables are configured.

```bash
cat << EOF > .env
# CHAT MODEL: Azure OpenAI via LiteLLM
CHAT_MODEL=azure/<your_deployment_name>
AZURE_API_BASE=https://<your_azure_base>.openai.azure.com/
AZURE_API_KEY=<your_azure_api_key>
AZURE_API_VERSION=<version>

# EMBEDDING MODEL: Using SiliconFlow via litellm_proxy
EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
LITELLM_PROXY_API_KEY=<your_siliconflow_api_key>
LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1
EOF
```

--------------------------------

### Configure Azure OpenAI with LiteLLM Python SDK

Source: https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html

This Python code configures environment variables for Azure OpenAI integration using the LiteLLM library. Ensure you replace the placeholder values with your specific Azure credentials and API version.

```python
from litellm import completion
import os

# Set Azure OpenAI environment variables
os.environ["AZURE_API_KEY"] = "<your_azure_api_key>"
os.environ["AZURE_API_BASE"] = "<your_azure_api_base>"
os.environ["AZURE_API_VERSION"] = "<version>"

```