### Install and Install Pre-commit Hooks Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/developer_guide/contributing.md Installs the pre-commit package and sets up the pre-commit hooks for the project. This should be done before committing code. ```bash pip install pre-commit pre-commit install ``` -------------------------------- ### Install QuantStats Library Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_GPM_Demo.ipynb Installs the quantstats library, which is required by the portfolio optimization environment for plotting graphs. ```bash # install quantstats # !pip install quantstats ``` -------------------------------- ### Install FinRL Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/README.md Install the FinRL library in editable mode. ```bash pip install -e . ``` -------------------------------- ### Install Ubuntu Dependencies Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md Installs necessary system packages for FinRL on Ubuntu. Ensure you have an active internet connection. ```bash sudo apt-get update && sudo apt-get install cmake libopenmpi-dev python3-dev zlib1g-dev libgl1-mesa-glx swig ``` -------------------------------- ### Install FinRL Library Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/MultipleStockTrading.md Installs the FinRL library from its GitHub repository. This is a prerequisite for using the library's functionalities. ```shell ## install finrl library !pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git ``` -------------------------------- ### FinRL Environment Setup Class Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/PortfolioAllocation.md The EnvSetup class provides methods to configure the trading environment for training, validation, and live trading. ```python class EnvSetup: """ Provides methods for retrieving daily stock data from Yahoo Finance API Attributes ---------- stock_dim: int number of unique stocks hmax : int maximum number of shares to trade initial_amount: int start money transaction_cost_pct : float transaction cost percentage per trade reward_scaling: float scaling factor for reward, good for training tech_indicator_list: list a list of technical indicator names (modified from config.py) Methods ------- create_env_training() create env class for training create_env_validation() create env class for validation create_env_trading() create env class for trading """ ``` -------------------------------- ### Install FinRL Dependencies (Windows) Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md Installs FinRL and its dependencies after cloning the repository on Windows. Navigate to the FinRL directory first. ```bash cd FinRL pip install . ``` -------------------------------- ### Install FinRL Library Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PortfolioOptimizationEnv_Demo.ipynb Installs the FinRL library and its dependencies, including swig. This is a prerequisite for using the FinRL environment. ```bash #!/bin/bash sudo apt install swig pip install git+https://github.com/AI4Finance-Foundation/FinRL.git ``` -------------------------------- ### Clone FinRL-Tutorials Repository Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md Download the FinRL-Tutorials repository to your local machine using git clone. This repository contains example notebooks and resources for using FinRL. ```bash git clone https://github.com/AI4Finance-Foundation/FinRL-Tutorials.git ``` -------------------------------- ### Custom Environment Setup Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PaperTrading_Demo.ipynb Defines the structure for a custom reinforcement learning environment, including state and action dimensions, and methods for resetting and stepping through the environment. ```python def __init__(self, gym_env_name, ...) -> None: '''the necessary env information when you design a custom env''' self.env_name = gym_env_name # the name of this env. self.state_dim = self.observation_space.shape[0] # feature number of state self.action_dim = self.action_space.shape[0] # feature number of action self.if_discrete = False # discrete action or continuous action def reset(self) -> np.ndarray: # reset the agent in env resetted_env, _ = self.env.reset() return resetted_env def step(self, action: np.ndarray) -> (np.ndarray, float, bool, dict): # We suggest that adjust action space to (-1, +1) when designing a custom env. state, reward, done, info_dict, _ = self.env.step(action * 2) return state.reshape(self.state_dim), float(reward), done, info_dict ``` -------------------------------- ### Install FinRL Library Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_GPM_Demo.ipynb Installs the FinRL library and its dependencies, including swig, for use in Google Colab environments. ```bash # install finrl library # !sudo apt install swig # !pip install git+https://github.com/AI4Finance-Foundation/FinRL.git ``` -------------------------------- ### Install System Packages with Homebrew Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md Install necessary system packages like cmake and openmpi using Homebrew. These are dependencies for certain FinRL functionalities. ```bash brew install cmake openmpi ``` -------------------------------- ### Install FinRL and Dependencies Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb Installs the FinRL library and its core dependencies using pip and apt-get. This includes essential system packages and Python libraries required for FinRL's functionality. ```python # ## install finrl library !pip install wrds !pip install swig !pip install -q condacolab import condacolab condacolab.install() !apt-get update -y -qq && apt-get install -y -qq cmake libopenmpi-dev python3-dev zlib1g-dev libgl1-mesa-glx swig !pip install git+https://github.com/AI4Finance-Foundation/FinRL.git ``` -------------------------------- ### Install FinRL and Dependencies Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/Stock_NeurIPS2018/Stock_NeurIPS2018_1_Data.ipynb Installs required packages for FinRL, including swig, wrds, pyportfolioopt, and the FinRL library itself from GitHub. ```python ## install required packages !pip install swig !pip install wrds !pip install pyportfolioopt ## install finrl library !pip install git+https://github.com/AI4Finance-Foundation/FinRL.git ``` -------------------------------- ### Replay Buffer Setup and Normalization Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/imitation_learning/Imitation_Sandbox.ipynb Creates a replay buffer, converts D4RL data into the required format, and flattens the state and next_state representations. It also normalizes the states if specified. ```python # replay_buffer = utils.ReplayBuffer(state_dim, action_dim) # replay_buffer.convert_D4RL(retail_train) # # flatten # replay_buffer.state = replay_buffer.state.reshape(replay_buffer.state.shape[0], -1) # replay_buffer.next_state = replay_buffer.next_state.reshape(replay_buffer.next_state.shape[0], -1) # if normalize: # mean,std = replay_buffer.normalize_states() # else: # mean,std = 0,1 ``` -------------------------------- ### Example Portfolio Vector Action Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/meta/env_portfolio_optimization/README.md Demonstrates a valid portfolio vector action for a portfolio with three stocks, showing the allocation percentages for cash and each stock. ```python W_{t} = [0.25, 0.4, 0.2, 0.15] ``` -------------------------------- ### Policy Initialization and Replay Buffer Setup Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/imitation_learning/Imitation_Sandbox.ipynb Initializes the TD3_BC policy with specified dimensions and hyperparameters. It also sets up a replay buffer and converts D4RL data, normalizing states if required. ```python print("---------------------------------------") print(f"Policy: {policy}, Env: {env}, Seed: {seed}") print("---------------------------------------") # Set seeds env.seed(seed) env.action_space.seed(seed) torch.manual_seed(seed) np.random.seed(seed) state_dim = env.observation_space.shape[0] * env.observation_space.shape[1] action_dim = env.action_space.shape[0] max_action = float(env.action_space.high[0]) kwargs = { "state_dim": state_dim, "action_dim": action_dim, "max_action": max_action, "discount": discount, "tau": tau, # TD3 "policy_noise": policy_noise * max_action, "noise_clip": noise_clip * max_action, "policy_freq": policy_freq, # TD3 + BC "alpha": alpha } # Initialize policy policy = TD3_BC.TD3_BC(**kwargs) # if load_model != "": # policy_file = file_name if load_model == "default" else load_model # policy.load(f"./models/{policy_file}") replay_buffer = utils.ReplayBuffer(state_dim, action_dim) replay_buffer.convert_D4RL(retail_train) # flatten replay_buffer.state = replay_buffer.state.reshape(replay_buffer.state.shape[0], -1) replay_buffer.next_state = replay_buffer.next_state.reshape(replay_buffer.next_state.shape[0], -1) if normalize: mean,std = replay_buffer.normalize_states() else: mean,std = 0,1 ``` -------------------------------- ### Import FinRL Library Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/Stock_NeurIPS2018/Stock_NeurIPS2018_2_Train.ipynb Imports the main finrl library after installation. ```python import finrl ``` -------------------------------- ### Prepare Test Data and Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/MultipleStockTrading.md Prepares the test dataset using the `data_split` function and initializes the testing environment with a specified turbulence threshold. This setup is crucial for evaluating the trading model's performance. ```python # test data test = data_split(dow_30, start='2019-01-01', end='2020-10-30') # testing env env_test = DummyVecEnv([lambda: StockEnvTrade(test, turbulence_threshold=insample_turbulence_threshold)]) obs_test = env_test.reset() ``` -------------------------------- ### Install FinRL Development Version Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md Install the unstable development version of FinRL directly from its GitHub repository using pip. This ensures you have the latest updates. ```bash pip install git+https://github.com/AI4Finance-Foundation/FinRL.git ``` -------------------------------- ### Install FinRL and Dependencies Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PaperTrading_Demo.ipynb Installs the FinRL library and necessary dependencies like condacolab, cmake, and openmpi for the trading environment. ```python ## install finrl library !pip install wrds !pip install swig !pip install -q condacolab import condacolab condacolab.install() !apt-get update -y -qq && apt-get install -y -qq cmake libopenmpi-dev python3-dev zlib1g-dev libgl1-mesa-glx swig !pip install git+https://github.com/AI4Finance-Foundation/FinRL.git ``` -------------------------------- ### Install FinRL Development Version Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/PortfolioAllocation.md Installs the unstable development version of FinRL. Use this command in a Jupyter notebook environment. ```python # Install the unstable development version in Jupyter notebook: !pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git ``` -------------------------------- ### Setup Trading Environment and Predict Account Value Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/SingleStockTrading.md Sets up a trading environment using `env_setup.create_env_trading` and then uses `DRLAgent.DRL_prediction` to predict the account value changes based on a given model and trading data. This is used for evaluating the performance of a trading strategy. ```python # create trading env env_trade, obs_trade = env_setup.create_env_trading(data = trade, env_class = StockEnvTrade, turbulence_threshold=250) ## make a prediction and get the account value change df_account_value = DRLAgent.DRL_prediction(model=model_sac, test_data = trade, test_env = env_trade, test_obs = obs_trade) ``` -------------------------------- ### Trading Environment Setup and Prediction Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/PortfolioAllocation.md This snippet demonstrates setting up a trading environment for portfolio allocation using the A2C model and performing predictions on a test dataset. It assumes an initial capital and a specific date range for trading. ```python trade = data_split(df,'2019-01-01', '2020-12-01') env_trade, obs_trade = env_setup.create_env_trading(data = trade, env_class = StockPortfolioEnv) df_daily_return, df_actions = DRLAgent.DRL_prediction(model=model_a2c, test_data = trade, test_env = env_trade, test_obs = obs_trade) ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/developer_guide/development_setup.md Create a new Conda environment named 'ai4finance' with Python 3.8, activate it, and install FinRL dependencies. ```bash cd ~/ai4finance conda create --name ai4finance python=3.8 conda activate ai4finance cd FinRL pip install -r requirements.txt ``` -------------------------------- ### Retail Weights Dataframe Example Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/imitation_learning/Weight_Initialization.ipynb Displays the structure and content of a dataframe after applying the retail weights (rank-based method). This shows the features and daily data points for various stocks. ```python Result: date open high low close volume tic day macd \ 0 2007-01-03 40.630 41.830 39.810 40.560 3873900 ADSK 2 0.000 0 2007-01-03 20.080 20.400 19.350 19.520 28350300 AMD 2 0.000 0 2007-01-03 27.460 27.980 27.330 19.536 64226000 CSCO 2 0.000 0 2007-01-03 30.170 30.290 28.200 28.500 487000 FSLR 2 0.000 0 2007-01-03 30.680 31.060 29.100 26.496 4734600 INTU 2 0.000 ... ... ... ... ... ... ... ... ... ... 2768 2017-12-29 85.630 86.050 85.500 80.353 18717400 MSFT 4 0.755 2768 2017-12-29 47.530 47.880 47.280 43.361 9750000 ORCL 4 -0.469 2768 2017-12-29 64.360 64.650 64.000 55.318 6631800 QCOM 4 0.254 2768 2017-12-29 104.580 105.080 104.420 90.749 2813300 TXN 4 1.784 2768 2017-12-29 19.080 19.150 19.010 15.257 2825800 WU 4 -0.096 boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma \ 0 42.034 39.856 100.000 66.667 100.000 40.560 40.560 0 20.037 19.273 100.000 -66.667 100.000 19.520 19.520 0 20.520 19.066 100.000 66.667 100.000 19.536 19.536 0 28.979 28.271 100.000 -66.667 100.000 28.500 28.500 0 27.201 26.159 100.000 -66.667 100.000 26.496 26.496 ... ... ... ... ... ... ... ... 2768 82.580 76.658 60.144 74.372 15.450 79.209 77.226 2768 46.547 42.377 42.781 -104.196 16.192 44.585 44.987 2768 56.595 55.253 56.700 -86.758 5.702 56.536 52.515 2768 93.422 81.992 68.523 122.502 47.232 86.948 84.777 2768 15.830 15.108 46.227 -149.472 34.911 15.524 15.640 moribvol 0 0.136 0 0.061 0 0.015 0 0.152 0 0.121 ... ... 2768 0.136 2768 0.015 2768 0.045 2768 0.121 2768 0.167 [30459 rows x 17 columns] ``` -------------------------------- ### Import FinRL and Supporting Packages Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/PortfolioAllocation.md Imports essential Python libraries for financial reinforcement learning, data manipulation, plotting, and environment setup. It also ensures that necessary directories for data storage, model training, and results are created. ```python # import packages import pandas as pd import numpy as np import matplotlib import matplotlib.pyplot as plt matplotlib.use('Agg') import datetime from finrl import config from finrl import config_tickers from finrl.marketdata.yahoodownloader import YahooDownloader from finrl.preprocessing.preprocessors import FeatureEngineer from finrl.preprocessing.data import data_split from finrl.env.environment import EnvSetup from finrl.env.EnvMultipleStock_train import StockEnvTrain from finrl.env.EnvMultipleStock_trade import StockEnvTrade from finrl.model.models import DRLAgent from finrl.trade.backtest import BackTestStats, BaselineStats, BackTestPlot, backtest_strat, baseline_strat from finrl.trade.backtest import backtest_strat, baseline_strat import os if not os.path.exists("./" + config.DATA_SAVE_DIR): os.makedirs("./" + config.DATA_SAVE_DIR) if not os.path.exists("./" + config.TRAINED_MODEL_DIR): os.makedirs("./" + config.TRAINED_MODEL_DIR) if not os.path.exists("./" + config.TENSORBOARD_LOG_DIR): os.makedirs("./" + config.TENSORBOARD_LOG_DIR) if not os.path.exists("./" + config.RESULTS_DIR): os.makedirs("./" + config.RESULTS_DIR) ``` -------------------------------- ### Install Homebrew Package Manager Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md Install Homebrew, a package manager for macOS, which is required for installing system dependencies for FinRL. ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` -------------------------------- ### Initialize FinRL Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/SingleStockTrading.md Demonstrates how to initialize an environment using the `EnvSetup` class. This involves specifying parameters like stock dimension, state space, maximum shares, initial amount, transaction cost, and technical indicators. ```python # Initialize env: env_setup = EnvSetup(stock_dim = stock_dimension, state_space = state_space, hmax = 100, initial_amount = 1000000, transaction_cost_pct = 0.001, tech_indicator_list = config.INDICATORS) env_train = env_setup.create_env_training(data = train, env_class = StockEnvTrain) ``` -------------------------------- ### Install Required Python Packages Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/SingleStockTrading.md Checks for and installs necessary Python packages for FinRL, including yfinance, pandas, matplotlib, stockstats, OpenAI gym, stable-baselines, and tensorflow. Ensure these are installed before proceeding. ```python import pkg_resources import pip installedPackages = {pkg.key for pkg in pkg_resources.working_set} required = {'yfinance', 'pandas', 'matplotlib', 'stockstats','stable-baselines','gym','tensorflow'} missing = required - installedPackages if missing: !pip install yfinance !pip install pandas !pip install matplotlib !pip install stockstats !pip install gym !pip install stable-baselines[mpi] !pip install tensorflow==1.15.4 ``` -------------------------------- ### Sample Retail Portfolio from Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/imitation_learning/Imitation_Sandbox.ipynb Samples a 'RETAIL' type portfolio from the initialized training environment. This is useful for generating baseline or reference portfolio data. ```python retail_train = StockPortfolioEnv.sample_from_env(i=0, env=e_train_gym, weight_type="RETAIL") ``` -------------------------------- ### Instantiate Portfolio Optimization Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PortfolioOptimizationEnv_Demo.ipynb Creates an instance of the PortfolioOptimizationEnv using prepared stock data. Key parameters include initial capital, commission fees, time window, and features to use. ```python environment = PortfolioOptimizationEnv( df_portfolio_train, initial_amount=100000, comission_fee_pct=0.0025, time_window=50, features=["close", "high", "low"], normalize_df=None ) ``` -------------------------------- ### Create and Activate Virtual Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/README.md Create a Python virtual environment and activate it for project dependencies. ```bash python3 -m venv venv source venv/bin/activate ``` -------------------------------- ### Instantiate Portfolio Optimization Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_GPM_Demo.ipynb Sets up the training and testing environments for portfolio optimization using the PortfolioOptimizationEnv class. It requires pre-processed dataframes and specifies initial investment, commission fees, time window, features, and relevant tickers. ```python df_portfolio = nasdaq_temporal[["day", "tic", "close", "high", "low"]] df_portfolio_train = df_portfolio[df_portfolio["day"] < 979] df_portfolio_test = df_portfolio[df_portfolio["day"] >= 979] environment_train = PortfolioOptimizationEnv( df_portfolio_train, initial_amount=100000, comission_fee_pct=0.0025, time_window=50, features=["close", "high", "low"], time_column="day", normalize_df=None, # dataframe is already normalized tics_in_portfolio=tics_in_portfolio ) environment_test = PortfolioOptimizationEnv( df_portfolio_test, initial_amount=100000, comission_fee_pct=0.0025, time_window=50, features=["close", "high", "low"], time_column="day", normalize_df=None, # dataframe is already normalized tics_in_portfolio=tics_in_portfolio ) ``` -------------------------------- ### Stock Portfolio Environment Initialization Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/PortfolioAllocation.md Initializes the StockPortfolioEnv, a custom gym environment for single stock trading simulations. It sets up the state and action spaces based on provided parameters. ```python import numpy as np import pandas as pd from gym.utils import seeding import gym from gym import spaces import matplotlib matplotlib.use('Agg') import matplotlib.pyplot as plt class StockPortfolioEnv(gym.Env): """A single stock trading environment for OpenAI gym Attributes ---------- df: DataFrame input data stock_dim : int number of unique stocks hmax : int maximum number of shares to trade initial_amount : int start money transaction_cost_pct: float transaction cost percentage per trade reward_scaling: float scaling factor for reward, good for training state_space: int the dimension of input features action_space: int equals stock dimension tech_indicator_list: list a list of technical indicator names turbulence_threshold: int a threshold to control risk aversion day: int an increment number to control date Methods ------- _sell_stock() perform sell action based on the sign of the action _buy_stock() perform buy action based on the sign of the action step() at each step the agent will return actions, then we will calculate the reward, and return the next observation. reset() reset the environment render() use render to return other functions save_asset_memory() return account value at each time step save_action_memory() return actions/positions at each time step """ metadata = {'render.modes': ['human']} def __init__(self, df, stock_dim, hmax, initial_amount, transaction_cost_pct, reward_scaling, state_space, action_space, tech_indicator_list, turbulence_threshold, lookback=252, day = 0): #super(StockEnv, self).__init__() #money = 10 , scope = 1 self.day = day self.lookback=lookback self.df = df self.stock_dim = stock_dim self.hmax = hmax self.initial_amount = initial_amount self.transaction_cost_pct =transaction_cost_pct self.reward_scaling = reward_scaling self.state_space = state_space self.action_space = action_space self.tech_indicator_list = tech_indicator_list # action_space normalization and shape is self.stock_dim self.action_space = spaces.Box(low = 0, high = 1,shape = (self.action_space,)) # Shape = (34, 30) # covariance matrix + technical indicators self.observation_space = spaces.Box(low=0, high=np.inf, shape = (self.state_space+len(self.tech_indicator_list), self.stock_dim)) # load data from a pandas dataframe self.data = self.df.loc[self.day,:] self.covs = self.data['cov_list'].values[0] self.state = np.append(np.array(self.covs), [self.data[tech].values.tolist() for tech in self.tech_indicator_list ], axis=0) self.terminal = False self.turbulence_threshold = turbulence_threshold # initalize state: inital portfolio return + individual stock return + individual weights self.portfolio_value = self.initial_amount ``` -------------------------------- ### Install box2d-py and Dependencies Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md Install the box2d-py package and its dependencies, including swig. This is required if you plan to use environments that rely on box2d. ```bash brew install swig pip install box2d-py pip install box2d pip install Box2D ``` -------------------------------- ### Troubleshoot box2d Installation Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/start/installation.md If you encounter specific errors like 'AttributeError: module ‘_Box2D’ has no attribute ‘RAND_LIMIT_swigconstant’', try installing alternative versions of the box2d package. ```bash pip install box2d box2d-kengz ``` -------------------------------- ### Instantiate Portfolio Optimization Environments Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PortfolioOptimizationEnv_Demo.ipynb Instantiates three separate PortfolioOptimizationEnv objects for different time periods (2020, 2021, 2022). Each environment is configured with specific dataframes, initial capital, commission fees, time window, and features. Use this to simulate trading strategies over distinct historical market conditions. ```python environment_2020 = PortfolioOptimizationEnv( df_portfolio_2020, initial_amount=100000, comission_fee_pct=0.0025, time_window=50, features=["close", "high", "low"], normalize_df=None ) environment_2021 = PortfolioOptimizationEnv( df_portfolio_2021, initial_amount=100000, comission_fee_pct=0.0025, time_window=50, features=["close", "high", "low"], normalize_df=None ) environment_2022 = PortfolioOptimizationEnv( df_portfolio_2022, initial_amount=100000, comission_fee_pct=0.0025, time_window=50, features=["close", "high", "low"], normalize_df=None ) ``` -------------------------------- ### Import Necessary Libraries Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/Stock_NeurIPS2018/Stock_NeurIPS2018_3_Backtest.ipynb Imports core libraries for data manipulation, plotting, and FinRL components. This setup is essential for all subsequent operations. ```python import sys import pandas as pd import numpy as np import matplotlib.pyplot as plt from finrl.meta.preprocessor.yahoodownloader import YahooDownloader from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv from finrl.agents.stablebaselines3.models import DRLAgent from stable_baselines3 import A2C, DDPG, PPO, SAC, TD3 %matplotlib inline from finrl.config import INDICATORS ``` -------------------------------- ### AgentBase Initialization Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PaperTrading_Demo.ipynb Initializes the base agent class with network dimensions, state and action dimensions, GPU ID, and configuration arguments. Sets up device, optimizers, and loss criterion. ```python class AgentBase: def __init__(self, net_dims: [int], state_dim: int, action_dim: int, gpu_id: int = 0, args: Config = Config()): self.state_dim = state_dim self.action_dim = action_dim self.gamma = args.gamma self.batch_size = args.batch_size self.repeat_times = args.repeat_times self.reward_scale = args.reward_scale self.soft_update_tau = args.soft_update_tau self.states = None # assert self.states == (1, state_dim) self.device = torch.device(f"cuda:{gpu_id}" if (torch.cuda.is_available() and (gpu_id >= 0)) else "cpu") act_class = getattr(self, "act_class", None) cri_class = getattr(self, "cri_class", None) self.act = self.act_target = act_class(net_dims, state_dim, action_dim).to(self.device) self.cri = self.cri_target = cri_class(net_dims, state_dim, action_dim).to(self.device) \ if cri_class else self.act self.act_optimizer = torch.optim.Adam(self.act.parameters(), args.learning_rate) self.cri_optimizer = torch.optim.Adam(self.cri.parameters(), args.learning_rate) \ if cri_class else self.act_optimizer self.criterion = torch.nn.SmoothL1Loss() ``` -------------------------------- ### Get DOW 30 Tickers Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/Stock_NeurIPS2018/Stock_NeurIPS2018_1_Data.ipynb Retrieves the list of tickers for the Dow 30 index. This list is used as input for data download. ```python config_tickers.DOW_30_TICKER ``` -------------------------------- ### Define Training and Trading Dates Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/Stock_NeurIPS2018/Stock_NeurIPS2018_1_Data.ipynb Sets the start and end dates for training and trading periods. These variables are crucial for time-series data splitting. ```python TRAIN_START_DATE = '2009-01-01' TRAIN_END_DATE = '2020-07-01' TRADE_START_DATE = '2020-07-01' TRADE_END_DATE = '2021-10-29' ``` -------------------------------- ### Configure Trading Environment Parameters Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/Stock_NeurIPS2018/Stock_NeurIPS2018_3_Backtest.ipynb Sets up the parameters for the StockTradingEnv, including trading costs, initial capital, and technical indicators. These parameters define the trading simulation's rules. ```python buy_cost_list = sell_cost_list = [0.001] * stock_dimension num_stock_shares = [0] * stock_dimension env_kwargs = { "hmax": 100, "initial_amount": 1000000, "num_stock_shares": num_stock_shares, "buy_cost_pct": buy_cost_list, "sell_cost_pct": sell_cost_list, "state_space": state_space, "stock_dim": stock_dimension, "tech_indicator_list": INDICATORS, "action_space": stock_dimension, "reward_scaling": 1e-4 } ``` -------------------------------- ### Example DataFrame Structure Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/meta/env_portfolio_optimization/README.md This illustrates the expected structure of the input dataframe for the POE environment, including date, price data, and ticker symbols. ```python date high low close tic 0 2020-12-23 0.157414 0.127420 0.136394 ADA-USD 1 2020-12-23 34.381519 30.074295 31.097898 BNB-USD 2 2020-12-23 24024.490234 22802.646484 23241.345703 BTC-USD 3 2020-12-23 0.004735 0.003640 0.003768 DOGE-USD 4 2020-12-23 637.122803 560.364258 583.714600 ETH-USD ... ... ... ... ... ... ``` -------------------------------- ### Import Required Packages Source: https://github.com/ai4finance-foundation/finrl/blob/master/docs/source/tutorial/Introduction/MultipleStockTrading.md Imports all necessary Python packages for data manipulation, visualization, FinRL components, and system operations. Ensure these are installed before running. ```python import pandas as pd import numpy as np import matplotlib import matplotlib.pyplot as plt # matplotlib.use('Agg') import datetime %matplotlib inline from finrl import config from finrl import config_tickers from finrl.meta.preprocessor.yahoodownloader import YahooDownloader from finrl.meta.preprocessor.preprocessors import FeatureEngineer, data_split from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv from finrl.agents.stablebaselines3.models import DRLAgent from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline from pprint import pprint import sys sys.path.append("../FinRL-Library") import itertools ``` -------------------------------- ### Environment Configuration for Portfolio Allocation Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/imitation_learning/Weight_Initialization.ipynb Sets up a gym-style portfolio allocation environment for agent training and comparison. It defines stock and feature dimensions, and initializes the training and trading environments with specified parameters. ```python train = train_data trade = trade_data stock_dimension = len(train.tic.unique()) state_space = stock_dimension tech_indicator_list = ['macd', 'rsi_30', 'cci_30', 'dx_30'] feature_dimension = len(tech_indicator_list) print(f"Stock Dimension: {stock_dimension}, State Space: {state_space}") print(f"Feature Dimension: {feature_dimension}") env_kwargs = { "hmax": 100, "initial_amount": 1000000, "transaction_cost_pct": 0, "state_space": state_space, "stock_dim": stock_dimension, "tech_indicator_list": tech_indicator_list, "action_space": stock_dimension, "reward_scaling": 1e-1 } e_train_gym = StockPortfolioEnv.StockPortfolioEnv(df = train, **env_kwargs) e_trade_gym = StockPortfolioEnv.StockPortfolioEnv(df = trade, **env_kwargs) ``` -------------------------------- ### DRL Agent Class Definition Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PaperTrading_Demo.ipynb Defines the DRLAgent class for managing DRL algorithms. It includes methods for getting a model, training it, and making predictions. ```python from __future__ import annotations import torch # from elegantrl.agents import AgentA2C MODELS = {"ppo": AgentPPO} OFF_POLICY_MODELS = ["ddpg", "td3", "sac"] ON_POLICY_MODELS = ["ppo"] # MODEL_KWARGS = {x: config.__dict__[f"{x.upper()}_PARAMS"] for x in MODELS.keys()} # # NOISE = { # "normal": NormalActionNoise, # "ornstein_uhlenbeck": OrnsteinUhlenbeckActionNoise, # } class DRLAgent: """Implementations of DRL algorithms Attributes ---------- env: gym environment class user-defined class Methods ------- get_model() setup DRL algorithms train_model() train DRL algorithms in a train dataset and output the trained model DRL_prediction() make a prediction in a test dataset and get results """ def __init__(self, env, price_array, tech_array, turbulence_array): self.env = env self.price_array = price_array self.tech_array = tech_array self.turbulence_array = turbulence_array def get_model(self, model_name, model_kwargs): env_config = { "price_array": self.price_array, "tech_array": self.tech_array, "turbulence_array": self.turbulence_array, "if_train": True, } environment = self.env(config=env_config) env_args = {'config': env_config, 'env_name': environment.env_name, 'state_dim': environment.state_dim, 'action_dim': environment.action_dim, 'if_discrete': False} agent = MODELS[model_name] if model_name not in MODELS: raise NotImplementedError("NotImplementedError") model = Config(agent_class=agent, env_class=self.env, env_args=env_args) model.if_off_policy = model_name in OFF_POLICY_MODELS if model_kwargs is not None: try: model.learning_rate = model_kwargs["learning_rate"] model.batch_size = model_kwargs["batch_size"] model.gamma = model_kwargs["gamma"] model.seed = model_kwargs["seed"] model.net_dims = model_kwargs["net_dimension"] model.target_step = model_kwargs["target_step"] model.eval_gap = model_kwargs["eval_gap"] model.eval_times = model_kwargs["eval_times"] except BaseException: raise ValueError( "Fail to read arguments, please check 'model_kwargs' input." ) return model def train_model(self, model, cwd, total_timesteps=5000): model.cwd = cwd model.break_step = total_timesteps train_agent(model) @staticmethod def DRL_prediction(model_name, cwd, net_dimension, environment): if model_name not in MODELS: raise NotImplementedError("NotImplementedError") agent_class = MODELS[model_name] environment.env_num = 1 agent = agent_class(net_dimension, environment.state_dim, environment.action_dim) actor = agent.act # load agent try: cwd = cwd + '/actor.pth' print(f"| load actor from: {cwd}") actor.load_state_dict(torch.load(cwd, map_location=lambda storage, loc: storage)) act = actor device = agent.device except BaseException: raise ValueError("Fail to load agent!") # test on the testing env _torch = torch state, _ = environment.reset() episode_returns = [] # the cumulative_return / initial_account episode_total_assets = [environment.initial_total_asset] with _torch.no_grad(): for i in range(environment.max_step): s_tensor = _torch.as_tensor((state,), device=device) a_tensor = act(s_tensor) # action_tanh = act.forward() action = ( a_tensor.detach().cpu().numpy()[0] ) # not need detach(), because with torch.no_grad() outside state, reward, done, _, _ = environment.step(action) total_asset = ( environment.amount + ( environment.price_ary[environment.day] * environment.stocks ).sum() ) episode_total_assets.append(total_asset) episode_return = total_asset / environment.initial_total_asset episode_returns.append(episode_return) if done: break print("Test Finished!") # return episode total_assets on testing data print("episode_return", episode_return) return episode_total_assets ``` -------------------------------- ### Get Unique Stock Tickers Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/imitation_learning/Stock_Selection.ipynb Extracts and displays the unique stock tickers from the processed large-cap tech dataset. This is useful for verifying the stock selection. ```python tech_largeCap_ret['ticker'].unique() ``` -------------------------------- ### Initialize and Configure PPO Agent Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/Stock_NeurIPS2018/Stock_NeurIPS2018_2_Train.ipynb Initializes a DRL agent for PPO, retrieves the PPO model with custom parameters, and sets up a logger for TensorBoard and CSV output. This is conditional on the `if_using_ppo` flag. ```python agent = DRLAgent(env = env_train) PPO_PARAMS = { "n_steps": 2048, "ent_coef": 0.01, "learning_rate": 0.00025, "batch_size": 128, } model_ppo = agent.get_model("ppo",model_kwargs = PPO_PARAMS) if if_using_ppo: # set up logger tmp_path = RESULTS_DIR + '/ppo' new_logger_ppo = configure(tmp_path, ["stdout", "csv", "tensorboard"]) # Set new logger model_ppo.set_logger(new_logger_ppo) ``` -------------------------------- ### Trade Dataframe Example Source: https://github.com/ai4finance-foundation/finrl/blob/master/finrl/applications/imitation_learning/Weight_Initialization.ipynb Illustrates the structure of the trade dataframe, which contains data for trading periods. This is typically used for backtesting or evaluating trading strategies. ```python Result: date open high low close volume tic day macd \ 0 2018-01-02 105.340 107.160 104.390 107.120 2040600 ADSK 1 -3.126 0 2018-01-02 10.420 11.020 10.340 10.980 44146300 AMD 1 -0.118 0 2018-01-02 38.670 38.950 38.430 33.136 20135700 CSCO 1 0.546 0 2018-01-02 67.840 70.500 67.840 70.430 1752200 FSLR 1 2.129 0 2018-01-02 158.220 158.850 157.010 152.836 1436100 INTU 1 1.228 ... ... ... ... ... ... ... ... ... ... 1007 2021-12-31 338.510 339.360 335.850 332.365 18000800 MSFT 4 2.333 1007 2021-12-31 88.050 88.100 87.180 85.402 5908200 ORCL 4 -1.536 1007 2021-12-31 183.310 185.150 182.600 177.918 4113300 QCOM 4 3.541 1007 2021-12-31 189.410 190.000 188.270 181.990 2813900 TXN 4 -0.804 1007 2021-12-31 18.170 18.250 17.840 16.777 4251100 WU 4 0.083 boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma \ 0 109.299 103.069 42.401 -46.823 16.989 111.624 116.601 0 11.174 9.591 47.857 16.873 1.262 10.656 11.574 0 33.333 31.723 68.414 104.513 41.566 32.173 30.622 0 74.356 61.051 63.063 70.593 22.722 65.429 59.588 0 155.434 147.894 59.470 59.491 11.443 150.864 147.513 ... ... ... ... ... ... ... ... 1007 343.209 315.177 55.069 41.028 10.666 330.022 322.471 1007 102.358 79.414 43.416 -97.508 15.966 90.801 92.164 1007 184.097 169.783 59.872 46.197 34.435 176.387 157.506 1007 192.027 176.924 47.910 -71.323 5.853 185.032 185.000 1007 17.348 16.069 47.645 64.486 9.153 16.269 17.031 moribvol 0 0.121 0 0.076 0 0.030 0 0.061 0 0.152 ... ... 1007 0.152 1007 0.030 1007 0.076 1007 0.136 1007 0.015 [11088 rows x 17 columns] ``` -------------------------------- ### Initialize Alpaca Paper Trading Environment Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PaperTrading_Demo.ipynb Initializes the paper trading environment using Alpaca with the ElegantRL library. Requires API keys and specifies trading parameters. ```python paper_trading_erl = AlpacaPaperTrading(ticker_list = DOW_30_TICKER, time_interval = '1Min', drl_lib = 'elegantrl', agent = 'ppo', cwd = './papertrading_erl_retrain', net_dim = ERL_PARAMS['net_dimension'], state_dim = state_dim, action_dim= action_dim, API_KEY = API_KEY, API_SECRET = API_SECRET, API_BASE_URL = API_BASE_URL, tech_indicator_list = INDICATORS, turbulence_thresh=30, max_stock=1e2) paper_trading_erl.run() ``` -------------------------------- ### AgentPPO Initialization Source: https://github.com/ai4finance-foundation/finrl/blob/master/examples/FinRL_PaperTrading_Demo.ipynb Initializes the PPO agent, inheriting from AgentBase. Sets specific PPO hyperparameters like ratio clipping, GAE lambda, and entropy lambda. ```python class AgentPPO(AgentBase): def __init__(self, net_dims: [int], state_dim: int, action_dim: int, gpu_id: int = 0, args: Config = Config()): self.if_off_policy = False self.act_class = getattr(self, "act_class", ActorPPO) self.cri_class = getattr(self, "cri_class", CriticPPO) AgentBase.__init__(self, net_dims, state_dim, action_dim, gpu_id, args) self.ratio_clip = getattr(args, "ratio_clip", 0.25) # `ratio.clamp(1 - clip, 1 + clip)` self.lambda_gae_adv = getattr(args, "lambda_gae_adv", 0.95) # could be 0.80~0.99 self.lambda_entropy = getattr(args, "lambda_entropy", 0.01) # could be 0.00~0.10 self.lambda_entropy = torch.tensor(self.lambda_entropy, dtype=torch.float32, device=self.device) ```