### Initializing QuantBook Object (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Creates an instance of the `QuantBook` class, which serves as the primary entry point for interacting with QuantConnect's research environment. This object is necessary for accessing data, running backtests, and performing research operations like fetching fundamental data or historical prices. ```python qb = QuantBook() ``` -------------------------------- ### Initializing QuantBook and Fetching Data C# Source: https://github.com/quantconnect/research/blob/master/Research2Production/CSharp/05 Stationary Processes and Z-Scores CSharp.ipynb Initializes the QuantBook research environment, selects a universe of liquid SP500 sector ETFs, and retrieves historical daily data for the specified symbols over a 360-day period. This sets up the data required for subsequent analysis. ```csharp // QuantBook C# Research Environment // For more information see https://www.quantconnect.com/docs/research/overview #load "../QuantConnect.csx" using QuantConnect.Data.UniverseSelection; var qb = new QuantBook(); var symbols = LiquidETFUniverse.SP500Sectors; var history = qb.History(symbols, 360, Resolution.Daily); ``` -------------------------------- ### Initializing QuantBook and Adding Symbols - Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/04 Kalman Filters and Pairs Trading.ipynb Sets up the QuantConnect `QuantBook` environment and adds the equity symbols 'VIA' and 'VIAB' for subsequent analysis. It also stores the `Symbol` objects in a list. This is the standard starting point for QuantConnect research scripts. ```python # QuantBook Analysis Tool # For more information see [https://www.quantconnect.com/docs/research/overview] import numpy as np from math import floor import matplotlib.pyplot as plt from KalmanFilter import KalmanFilter qb = QuantBook() symbols = [qb.AddEquity(x).Symbol for x in ['VIA', 'VIAB']] ``` -------------------------------- ### Initializing QuantConnect Research Environment in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Sets up the QuantConnect QuantBook environment, enables inline plotting, and imports core QuantConnect libraries (`QuantConnect`, `QuantConnect.Data`, `QuantConnect.Jupyter`, `QuantConnect.Indicators`), standard Python libraries (`datetime`, `timedelta`, `matplotlib.pyplot`, `pandas`, `numpy`), and the SciPy optimization module (`scipy.optimize.minimize`). It creates a `QuantBook` instance named `qb` for data access and backtesting capabilities. ```python %matplotlib inline from clr import AddReference AddReference("System") AddReference("QuantConnect.Common") AddReference("QuantConnect.Jupyter") AddReference("QuantConnect.Indicators") from System import * from QuantConnect import * from QuantConnect.Data.Market import TradeBar, QuoteBar from QuantConnect.Jupyter import * from QuantConnect.Indicators import * from datetime import datetime, timedelta import matplotlib.pyplot as plt import pandas as pd import numpy as np from scipy.optimize import minimize import matplotlib.pyplot as plt import matplotlib.cm as cm qb = QuantBook() ``` -------------------------------- ### Displaying Optimal Portfolio Metrics DataFrame in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Simply outputs the contents of the pandas DataFrame `df` created in the previous step. In a Jupyter environment, this would typically render the DataFrame in a formatted table. ```python df ``` -------------------------------- ### Fetching Historical Equity Data with QuantConnect in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Sets a random seed for reproducibility. Defines a list of equity symbols. Iterates through the symbols, adds each one to the `QuantBook` using `qb.AddEquity` with daily resolution, fetches 500 days of historical daily data using `qb.History`, extracts the 'close' price, and stores it in a dictionary. Finally, it converts the dictionary of prices into a pandas DataFrame `df_price`. ```python np.random.seed(123) symbols = ['CCE','AAP','AAPL','GOOG','IBM','AMZN'] data = {} for i in symbols: qb.AddEquity(i, Resolution.Daily) history = qb.History([i], 500, Resolution.Daily) data[i] = history.loc[i]['close'] df_price = pd.DataFrame(data,columns=data.keys()) ``` -------------------------------- ### Importing Matplotlib Library (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Imports the `matplotlib.pyplot` module, commonly used for creating visualizations, and assigns it the alias `plt` for easier access. This is required for generating plots of financial data in the research environment. ```python import matplotlib.pyplot as plt ``` -------------------------------- ### Calculating Returns, Mean, and Std Dev C# Source: https://github.com/quantconnect/research/blob/master/Research2Production/CSharp/05 Stationary Processes and Z-Scores CSharp.ipynb Iterates through the historical data slice by slice to calculate the daily percentage return for each symbol. Subsequently, it computes the mean and standard deviation of these daily returns for each symbol using basic statistical methods and LINQ extensions. ```csharp // Calculates the % returns of our symbols over the historical data period var returns = new Dictionary>(); var last = new Dictionary(); foreach(var slice in history){ foreach(var symbol in slice.Bars.Keys){ if(!returns.ContainsKey(symbol)){ returns.Add(symbol, new List()); last.Add(symbol, (decimal)slice.Bars[symbol].Close); } var change = (decimal) ((decimal)slice.Bars[symbol].Close - last[symbol])/last[symbol]; returns[symbol].Add(change); } } var means = new Dictionary(); // Calculate mean price over 30 day period for each symbol foreach(var symbol in returns.Keys){ if(!means.ContainsKey(symbol)){ var avg = returns[symbol].Average(); means.Add(symbol, avg); } } var std = new Dictionary(); // Calculate standard deviation of prices for each symbol foreach(var symbol in returns.Keys){ if(!std.ContainsKey(symbol)){ var average = means[symbol]; var sumOfSquaresOfDifferences = returns[symbol].Select(val => (val - average) * (val - average)).Sum(); var sd = (Decimal)Math.Sqrt((double) (sumOfSquaresOfDifferences / returns[symbol].Count())); std.Add(symbol, sd); } } ``` -------------------------------- ### Structuring Optimal Portfolio Metrics in pandas DataFrame in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Creates a pandas DataFrame `df` to organize the return and volatility of the optimal portfolio found in `opt_portfolio`. It uses a dictionary where keys 'return' and 'volatility' correspond to the second and third elements of the `opt_portfolio` tuple, and sets the index to 'optimal portfolio'. ```python # print out the return, volatility, Sharpe ratio and weights of the optimal portfolio df = pd.DataFrame({'return':opt_portfolio[1], 'volatility':opt_portfolio[2]}, index=['optimal portfolio']) ``` -------------------------------- ### Instantiating Portfolio Optimization Class and Finding Optimal Weights in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Creates an instance `a` of the `PortfolioOptimization` class, passing the calculated `log_return` DataFrame, a risk-free rate of 0, and the number of assets derived from the `symbols` list. It then calls the `opt_portfolio()` method on the instance `a` to calculate the optimal portfolio based on maximum Sharpe ratio and stores the result in `opt_portfolio`. Finally, it accesses and displays the optimal weights (the first element of the `opt_portfolio` tuple). ```python # create the portfolio object 'a' with given symbols a = PortfolioOptimization(log_return, 0, len(symbols)) opt_portfolio = a.opt_portfolio() opt_portfolio[0] ``` -------------------------------- ### Implementing Mean-Variance Portfolio Optimization in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Defines the `PortfolioOptimization` class, which takes log returns (pandas DataFrame), risk-free rate (float), and number of assets (int) as input. It includes methods to calculate portfolio annual return and volatility, perform Monte Carlo simulation for feasible portfolios, find the optimal portfolio (maximizing Sharpe ratio) and the minimum variance portfolio using `scipy.optimize.minimize`, calculate the efficient frontier, and plot the results including the Capital Market Line. ```python class PortfolioOptimization: """ Description: This class shows how to set up a basic asset allocation problem that uses mean-variance portfolio optimization to estimate efficient portfolio and plot the efficient frontier and capital market line Args: log_return(pandas.DataFrame): The log return for assets in portfolio index: date columns: symbols value: log return series risk_free_rate(float): The risk free rate num_assets(int): The number of assets in portfolio """ def __init__(self, log_return, risk_free_rate, num_assets): self.log_return = log_return self.risk_free_rate = risk_free_rate self.n = num_assets def annual_port_return(self, weights): ''' calculate the annual return of portfolio ''' return np.sum(self.log_return.mean() * weights) * 252 def annual_port_vol(self, weights): ''' calculate the annual volatility of portfolio ''' return np.sqrt(np.dot(weights.T, np.dot(self.log_return.cov() * 252, weights))) def mc_mean_var(self): ''' apply monte carlo method to search for the feasible region of return ''' returns = [] vols = [] for i in range(5000): weights = np.random.rand(self.n) weights /= np.sum(weights) returns.append(self.annual_port_return(weights)) vols.append(self.annual_port_vol(weights)) return returns, vols def min_func(self, weights): return - self.annual_port_return(weights) / self.annual_port_vol(weights) def opt_portfolio(self): ''' maximize the sharpe ratio to find the optimal weights ''' cons = ({'type': 'eq', 'fun': lambda x: np.sum(x) - 1}) bnds = tuple((0, 1) for x in range(self.n)) opt = minimize(self.min_func, np.array(self.n * [1. / self.n]), method='SLSQP', bounds=bnds, constraints=cons) opt_weights = opt['x'] opt_return = self.annual_port_return(opt_weights) opt_volatility = self.annual_port_vol(opt_weights) return opt_weights, opt_return, opt_volatility def min_var_portfolio(self): ''' find the portfolio with minimum volatility ''' cons = ({'type': 'eq', 'fun': lambda x: np.sum(x) - 1}) bnds = tuple((0, 1) for x in range(self.n)) opt = minimize(self.annual_port_vol, np.array(self.n * [1. / self.n]), method='SLSQP', bounds=bnds, constraints=cons) min_vol_weights = opt['x'] min_vol_return = self.annual_port_return(min_vol_weights) min_volatility = self.annual_port_vol(min_vol_weights) return min_vol_weights, min_vol_return, min_volatility def efficient_frontier(self, mc_returns): ''' calculate the efficient frontier ''' target_return = np.linspace(min(mc_returns), max(mc_returns) + 0.05, 100) target_vol = [] bnds = tuple((0, 1) for x in range(self.n)) for i in target_return: cons = ({'type': 'eq', 'fun': lambda x: self.annual_port_return(x) - i}, {'type': 'eq', 'fun': lambda x: np.sum(x) - 1}) opt = minimize(self.annual_port_vol, np.array(self.n * [1. / self.n]), method='SLSQP', bounds=bnds, constraints=cons) target_vol.append(opt['fun']) return target_vol, target_return def plot(self): mc_returns, mc_vols = self.mc_mean_var() target_vol, target_return = self.efficient_frontier(mc_returns) opt_weights, opt_return, opt_volatility = self.opt_portfolio() min_vol_weights, min_vol_return, min_volatility = self.min_var_portfolio() plt.figure(figsize=(15, 8)) # plot the possible mean-variance portfolios with monte carlo simulation excess_return = [i - self.risk_free_rate for i in mc_returns] plt.scatter(mc_vols, mc_returns, c=np.array(excess_return) / np.array(mc_vols), cmap=cm.jet, marker='.') plt.grid() plt.xlabel('standard deviation(annual)') plt.ylabel('expected return(annual)') plt.colorbar(label='Sharpe ratio') # plot the efficient frontier plt.scatter(target_vol, target_return, c=np.array(target_return) / np.array(target_vol), marker='.', cmap=cm.jet) # mark the optimal portfolio with green star plt.scatter(opt_volatility, opt_return, marker='*', s=200, c='g', label = 'optimal portfolio') # mark the min volatility portfolio with purple star plt.scatter(min_volatility, min_vol_return, marker='*', s=200, c='m', label = 'min volatility portfolio') # plot the capital market line with black cml_x = np.linspace(0.0, 0.3) cml_slope = (opt_return - self.risk_free_rate) / opt_volatility plt.plot(cml_x, self.risk_free_rate + cml_slope * cml_x, lw=1.5, c='k', label = 'capital market line') plt.legend() ``` -------------------------------- ### Defining Tickers and Adding Equities to QuantBook (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Defines a Python list of stock ticker symbols for several large airline companies. It then iterates through this list, adding each ticker as an equity security to the `QuantBook` object with daily resolution and storing the resulting `Symbol` objects in a new list for subsequent data retrieval. ```python tickers = [ "ALK", # Alaska Air Group, Inc. "AAL", # American Airlines Group, Inc. "DAL", # Delta Air Lines, Inc. "LUV", # Southwest Airlines Company "UAL", # United Air Lines "ALGT", # Allegiant Travel Company "SKYW" # SkyWest, Inc. ] symbols = [qb.AddEquity(ticker, Resolution.Daily).Symbol for ticker in tickers] ``` -------------------------------- ### Calculating Log Returns and Plotting Histograms in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Calculates daily log returns by taking the natural logarithm of the ratio of current price to previous price for the `df_price` DataFrame and drops the initial row which contains NaN values. It then generates histograms for the log return series of each stock using the pandas `.hist()` method, specifying the number of bins and figure size. ```python # calculate the log return series for each stock log_return = np.log(df_price / df_price.shift(1)).dropna() # plot the histogram of stocks log_return.hist(bins=50, figsize=(15, 12)) ``` -------------------------------- ### Plotting PE Ratios Over Time (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Generates a line plot of the PE ratios for each symbol over the specified time period using the DataFrame's built-in `plot` method. It sets the plot size and title, labels the axes using `matplotlib.pyplot`, and displays the plot. This requires the `pe_ratios` DataFrame and `matplotlib` to be imported. ```python pe_ratios.plot(figsize=(16, 8), title="PE Ratio Over Time") plt.xlabel("Time") plt.ylabel("PE Ratio") plt.show() ``` -------------------------------- ### Plotting Portfolio Optimization Results in Python Source: https://github.com/quantconnect/research/blob/master/Analysis/03 Mean-Variance Portfolio Optimization .ipynb Calls the `plot()` method of the `PortfolioOptimization` instance `a`. This method performs Monte Carlo simulation, calculates the efficient frontier, finds the optimal and minimum variance portfolios, and generates a matplotlib plot showing the feasible region, efficient frontier, optimal portfolio, minimum variance portfolio, and the Capital Market Line. ```python a.plot() ``` -------------------------------- ### Calculating and Plotting Cumulative Returns (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Calculates the cumulative returns for the historical closing prices of the two selected securities. It computes percentage changes, calculates the cumulative product, and then plots these cumulative returns over time, setting the title and ylabel using matplotlib. ```python returns_over_time = ((history.pct_change()[1:] + 1).cumprod() - 1) returns_over_time.plot(figsize=(16, 8), title="Returns Over Time") plt.ylabel("Return") plt.show() ``` -------------------------------- ### Fetching PE Ratios with GetFundamental (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Retrieves the "ValuationRatios.PERatio" fundamental data for the list of defined equity symbols (`symbols`) over the specified date range (January 1, 2014, to January 1, 2015) using the `QuantBook.GetFundamental` method. The retrieved data is stored in a pandas DataFrame-like object for further analysis. ```python pe_ratios = qb.GetFundamental(symbols, "ValuationRatios.PERatio", datetime(2014, 1, 1), datetime(2015, 1, 1)) pe_ratios ``` -------------------------------- ### Selecting Symbols with Highest/Lowest Average PE (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Selects the symbols corresponding to the highest and lowest average PE ratios from the sorted Series index (`sorted_by_mean_pe.index`). It uses `qb.Symbol()` to convert the ticker strings from the index back into QuantConnect `Symbol` objects, storing the highest in `highest_avg_pe` and the lowest in `lowest_avg_pe` for further analysis. ```python highest_avg_pe = qb.Symbol(sorted_by_mean_pe.index[-1]) lowest_avg_pe = qb.Symbol(sorted_by_mean_pe.index[0]) ``` -------------------------------- ### Calculating and Sorting Mean PE Ratios (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Calculates the mean PE ratio for each security across the entire date range using the `mean()` method on the `pe_ratios` DataFrame. It then sorts these mean values in ascending order using `sort_values()`, returning a pandas Series where the index is the symbol string and the values are the mean PE ratios. ```python sorted_by_mean_pe = pe_ratios.mean().sort_values() sorted_by_mean_pe ``` -------------------------------- ### Calculating and Printing Z-Scores C# Source: https://github.com/quantconnect/research/blob/master/Research2Production/CSharp/05 Stationary Processes and Z-Scores CSharp.ipynb Computes the Z-score for each daily return using the previously calculated mean and standard deviation for the respective symbol. Finally, it iterates through the calculated Z-scores and prints the symbol along with the total count of Z-scores calculated for it. ```csharp var ZScores = new Dictionary>(); // Calculate the Z-Score for each return in the time series foreach(var symbol in returns.Keys){ if(!ZScores.ContainsKey(symbol)){ ZScores.Add(symbol, new List()); } foreach(var ret in returns[symbol]){ var score = (decimal)((ret - means[symbol])/std[symbol]); ZScores[symbol].Add(score); } } foreach(var symbol in ZScores.Keys){ Console.WriteLine(symbol + ", " + ZScores[symbol].Count); } ``` -------------------------------- ### Fetching Historical Closing Prices (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Stock-Picking-With-Fundamental-Data.ipynb Retrieves historical daily closing prices (`.close`) for the two selected `Symbol` objects (`highest_avg_pe`, `lowest_avg_pe`) over a specified date range (January 1, 2009, to January 1, 2015) using `QuantBook.History`. It then uses `unstack(level=0)` to reshape the resulting DataFrame for easier plotting of returns. ```python history = qb.History([highest_avg_pe, lowest_avg_pe], datetime(2009, 1, 1), datetime(2015, 1, 1), Resolution.Daily).close.unstack(level=0) history ``` -------------------------------- ### Clearing ObjectStore Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-TensorFlow.ipynb Calls the `clear_ObjectStore` helper function to remove all saved items (the graph and weights) from the QuantConnect ObjectStore, cleaning up the environment after the example run. ```python clear_ObjectStore() ``` -------------------------------- ### Executing QuantConnect ObjectStore Cleanup Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-Keras.ipynb Calls the `clear_ObjectStore` function to remove all objects currently stored in the QuantConnect ObjectStore. This is typically used for cleaning up the storage after running the example or during development. ```Python clear_ObjectStore() ``` -------------------------------- ### Setting up QuantBook Environment and Imports (Python) Source: https://github.com/quantconnect/research/blob/master/Analysis/05 Pairs Trading Strategy Based on Cointegration.ipynb Imports necessary libraries for the QuantConnect research environment, including system, common, jupyter, and indicators. Initializes a QuantBook instance, which is required for accessing historical data and interacting with the LEAN engine in research notebooks. ```python %matplotlib inline # Imports from clr import AddReference AddReference("System") AddReference("QuantConnect.Common") AddReference("QuantConnect.Jupyter") AddReference("QuantConnect.Indicators") from System import * from QuantConnect import * from QuantConnect.Data.Market import TradeBar, QuoteBar from QuantConnect.Jupyter import * from QuantConnect.Indicators import * from datetime import datetime, timedelta import matplotlib.pyplot as plt import pandas as pd # Create an instance qb = QuantBook() ``` -------------------------------- ### Initialize QuantBook and Import Libraries - Python Source: https://github.com/quantconnect/research/blob/master/Topical/20200507_hopevfear_research.ipynb Initializes the QuantConnect research environment by creating a QuantBook instance and importing necessary libraries for data handling, numerical operations, date/time manipulation, and custom data sources like TiingoNews. ```python from QuantConnect.Data.Custom.Tiingo import * import pandas as pd import numpy as np import scipy from datetime import datetime, timedelta qb = QuantBook() ``` -------------------------------- ### Fetching SPY Historical Data with QuantBook Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-Keras.ipynb Initializes a QuantBook object, adds the SPY equity to the universe, and retrieves 360 days of daily historical data for the SPY ticker using the QuantConnect history method. The data is then filtered to get only the SPY history. ```Python qb = QuantBook() spy = qb.AddEquity('SPY') history = qb.History(qb.Securities.Keys, 360, Resolution.Daily) spy_hist = history.loc['SPY'] spy_hist ``` -------------------------------- ### Initializing QuantBook and Data - Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/06 Principle Compenent Analysis.ipynb This snippet initializes the QuantConnect research environment (`QuantBook`) and loads the required financial data source. It imports necessary libraries for numerical operations, statistical modeling, machine learning (decomposition, ensembles, linear models, model selection), and QuantConnect-specific data types, adding the daily US Treasury Yield Curve Rate data. ```Python import numpy as np import statsmodels.api as sm from sklearn.decomposition import PCA from sklearn.ensemble import RandomForestRegressor from sklearn.linear_model import Ridge, LinearRegression, LogisticRegression, HuberRegressor, Lasso from sklearn.model_selection import cross_val_score, GridSearchCV # Import the Liquid ETF Universe helper methods from QuantConnect.Data.UniverseSelection import * from QuantConnect.Data.Custom.USTreasury import * # Initialize QuantBook and the US Treasuries ETFs qb = QuantBook() yieldCurve = qb.AddData(USTreasuryYieldCurveRate, "USTYCR", Resolution.Daily).Symbol ``` -------------------------------- ### Initializing QuantBook Environment - C# Source: https://github.com/quantconnect/research/blob/master/Research2Production/CSharp/04 Kalman Filters and Pairs Trading CSharp.ipynb This snippet initializes the QuantConnect C# Research environment by creating a `QuantBook` instance. It then adds two equity symbols, 'VIA' and 'VIAB', with daily resolution to the book for data access and analysis. The `Symbol` objects are stored for convenient referencing in subsequent code. ```C# // QuantBook C# Research Environment // For more information see https://www.quantconnect.com/docs/research/overview #load "../QuantConnect.csx" var qb = new QuantBook(); var via = qb.AddEquity("VIA", Resolution.Daily).Symbol; var viab = qb.AddEquity("VIAB", Resolution.Daily).Symbol; ``` -------------------------------- ### Initialize QuantBook and Fetch Data - Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/07 Hidden Markov Models.ipynb This snippet initializes the QuantBook research environment, adds the SPY equity, fetches 500 hours of historical data, and calculates the percentage change in closing prices, dropping any missing values. ```Python # Import necessary libraries and get historical data import numpy as np from hmmlearn.hmm import GaussianHMM qb = QuantBook() symbol = qb.AddEquity('SPY', Resolution.Daily).Symbol # Fetch history and returns history = qb.History(symbol, 500, Resolution.Hour) returns = history.close.pct_change().dropna() ``` -------------------------------- ### Initializing QuantBook and Fetching Historical Data C# Source: https://github.com/quantconnect/research/blob/master/Research2Production/CSharp/01 Mean Reversion CSharp.ipynb This snippet initializes the QuantBook environment, defines a list of target assets, adds them to the security universe with minute resolution data, fetches the last 30 days of daily historical data, and extracts the close prices for each asset over the historical period. ```C# // QuantBook C# Research Environment // For more information see https://www.quantconnect.com/docs/research/overview #load "../QuantConnect.csx" // Initialize QuantBook var qb = new QuantBook(); var assets = new List() {"SHY", "TLT", "SHV", "TLH", "EDV", "BIL", "SPTL", "TBT", "TMF", "TMV", "TBF", "VGSH", "VGIT", "VGLT", "SCHO", "SCHR", "SPTS", "GOVT"}; // Subscribed to data for given assets foreach(var ticker in assets){ qb.AddEquity(ticker, Resolution.Minute); } // Request past 30 days of historical price data for given assets var history = qb.History(qb.Securities.Keys, 30, Resolution.Daily); var closes = new Dictionary>(); // extract close prices for each Symbol from Slice data foreach(var slice in history){ foreach(var symbol in slice.Keys){ if(!closes.ContainsKey(symbol)){ closes.Add(symbol, new List()); } closes[symbol].Add(slice.Bars[symbol].Close); } } ``` -------------------------------- ### Initializing QuantBook and Fetching History (C#) Source: https://github.com/quantconnect/research/blob/master/Research2Production/CSharp/03 Uncorrelated Assets CSharp.ipynb This snippet sets up the QuantBook research environment by creating a QuantBook instance, adds a predefined list of equity tickers, and retrieves historical hour-resolution data for these securities over the last 150 periods using the `History` method. It requires the `QuantConnect.csx` script and the `MathNet.Numerics.Statistics` library. ```C# // QuantBook C# Research Environment // For more information see https://www.quantconnect.com/docs/research/overview #load "../QuantConnect.csx" using MathNet.Numerics.Statistics; var qb = new QuantBook(); var tickers = new List {"SQQQ", "TQQQ", "TVIX", "VIXY", "SPLV", "SVXY", "UVXY", "EEMV", "EFAV", "USMV"}; foreach(var ticker in tickers){ qb.AddEquity(ticker, Resolution.Minute); } // Fetch history var history = qb.History(qb.Securities.Keys, 150, Resolution.Hour); ``` -------------------------------- ### Initializing QuantBook and Universe Selection - Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/05 Stationary Processes and Z-Scores.ipynb Imports necessary modules including custom stationarity/z-score functions and QuantConnect's universe helpers. Initializes the QuantBook environment and defines a list of symbols representing S&P 500 sector ETFs using `LiquidETFUniverse` for subsequent data fetching. ```python # Import our custom functions from StationarityAndZScores import * # Import the Liquid ETF Universe helper methods from QuantConnect.Data.UniverseSelection import * # Initialize QuantBook and the Sector ETFs qb = QuantBook() symbols = [x for x in LiquidETFUniverse.SP500Sectors] ``` -------------------------------- ### Importing Libraries and Initializing QuantBook Source: https://github.com/quantconnect/research/blob/master/Analysis/04 EMA Cross Strategy Based on VXX.ipynb This snippet imports necessary Python libraries including QuantConnect components for research and data handling, System for basic operations, and standard libraries like pandas and matplotlib. It also initializes a QuantBook instance `qb` to interact with QuantConnect data and indicators. ```python %matplotlib inline # Imports from clr import AddReference AddReference("System") AddReference("QuantConnect.Common") AddReference("QuantConnect.Jupyter") AddReference("QuantConnect.Indicators") from System import * from QuantConnect import * from QuantConnect.Data.Market import TradeBar, QuoteBar from QuantConnect.Jupyter import * from QuantConnect.Indicators import * from datetime import datetime, timedelta import matplotlib.pyplot as plt import pandas as pd import numpy as np from math import ceil, floor # Create an instance qb = QuantBook() plt.style.use('seaborn-whitegrid') ``` -------------------------------- ### Getting Top Uncorrelated Assets Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/03 Uncorrelated Assets.ipynb Calls the previously defined `GetUncorrelatedAssets` function using the calculated `returns` DataFrame. It requests the top 5 assets with the lowest correlation rank. The result, a list of (symbol, rank) tuples, is stored in the `selected` variable. Requires the `GetUncorrelatedAssets` function and the `returns` DataFrame from the previous step. ```python selected = GetUncorrelatedAssets(returns, 5) selected ``` -------------------------------- ### Initializing and Fitting PCA Model - Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/06 Principle Compenent Analysis.ipynb This code initializes a Principal Component Analysis (PCA) model from scikit-learn, specifying that it should retain the minimum number of components required to explain at least 95% of the data's variance. It then fits this PCA model to the `training` data set, computing the principal components and their corresponding explained variance ratios. ```Python # Initialize the PCA model pca = PCA(n_components=0.95) # Forces it to explain >99% of variance # Fit the PCA model pca.fit(training) print(f'PCA Explained Variance: {pca.explained_variance_ratio_}\n') print(f'PCA No. Components: {pca.n_components_}\n') ``` -------------------------------- ### Importing Libraries Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-TensorFlow.ipynb Imports necessary libraries for TensorFlow model building, data splitting, and JSON serialization/deserialization, including specific QuantConnect helper classes for handling Protocol Buffer messages. ```python import tensorflow as tf from sklearn.model_selection import train_test_split import json5 from google.protobuf import json_format ``` -------------------------------- ### Fetching Data and Running Kalman Filter - Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/04 Kalman Filters and Pairs Trading.ipynb Initializes the `KalmanFilter`, retrieves historical daily closing prices for the selected symbols from 2019-01-01 to 2019-01-11 using `qb.History`. It then iterates through the daily price data, calling the `kf.update()` method with the prices for 'VIA' and 'VIAB' and prints the forecast error, prediction standard deviation, and calculated hedge quantity. ```python kf = KalmanFilter() history = qb.History(qb.Securities.Keys, datetime(2019, 1, 1), datetime(2019, 1, 11), Resolution.Daily) prices = history.unstack(level=1).close.transpose() for index, row in prices.iterrows(): via = row.loc[str(symbols[0].ID)] viab = row.loc[str(symbols[1].ID)] forecast_error, prediction_std_dev, hedge_quantity = kf.update(via, viab) print(f'{forecast_error} :: {prediction_std_dev} :: {hedge_quantity}') ``` -------------------------------- ### Fetch News History and Count Sentiment Words - Python Source: https://github.com/quantconnect/research/blob/master/Topical/20200507_hopevfear_research.ipynb Iterates through the defined S&P500 tickers, adds TiingoNews data for each, fetches 127 days of daily news history, and counts the occurrences of fearful and hopeful words in each article's description using the `count_instances` function. It then resamples the counts to get daily sums per ticker and stores these sums in lists, handling potential errors for tickers with no data. ```python # Extract history and get word count of fearful and hopeful words, per day, per ticker hopeful_word_sums = [] fearful_word_sums = [] tickers_not_in_data = [] # Note: This takes 45 minutes to run for ticker in sp500: symbol = qb.AddEquity(ticker).Symbol news = qb.AddData(TiingoNews, symbol).Symbol history = qb.History(TiingoNews, news, timedelta(days=127), Resolution.Daily) try: description = history.reset_index(level=0)['description'] hopeful_word_count = description.apply(lambda x: count_instances(hopeful_words, x.split(' '))) hopeful_word_count_daily = hopeful_word_count.resample('D').sum() hopeful_word_sums.append(hopeful_word_count_daily) fearful_word_count = description.apply(lambda x: count_instances(fearful_words, x.split(' '))) fearful_word_count_daily = fearful_word_count.resample('D').sum() fearful_word_sums.append(fearful_word_count_daily) except: tickers_not_in_data.append(ticker) continue ``` -------------------------------- ### Training TensorFlow Model Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-TensorFlow.ipynb Configures the training process by defining the mean squared error loss function and using the Adam optimizer. It initializes all global variables and then runs the training loop over multiple epochs, processing data in batches. ```python loss = tf.reduce_mean(tf.squared_difference(output, Y)) optimizer = tf.train.AdamOptimizer().minimize(loss) sess.run(tf.global_variables_initializer()) batch_size = len(y_train) // 10 epochs = 20 for _ in range(epochs): for i in range(0, len(y_train) // batch_size): start = i * batch_size batch_x = X_train[start:start + batch_size] batch_y = y_train[start:start + batch_size] sess.run(optimizer, feed_dict={X: batch_x, Y: batch_y}) ``` -------------------------------- ### Fetching GLD and BAR Data and Calculating Returns QuantConnect Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Plotting-Price-Data-Using-Plotly.ipynb Adds GLD and BAR equities to the QuantBook and retrieves 300 days of daily historical data for both. It then calculates the daily percentage change of the 'close' price for each ETF, dropping the initial NaN value, to obtain their daily returns. ```Python qb.AddEquity('GLD') qb.AddEquity('BAR') gold_etf_hist = qb.History(['GLD', 'BAR'], 300, Resolution.Daily) gld_ret = gold_etf_hist.loc['GLD']['close'].pct_change()[1:] bar_ret = gold_etf_hist.loc['BAR']['close'].pct_change()[1:] ``` -------------------------------- ### Gathering and Preparing Historical Data with QuantConnect Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Plotting-Price-Data-Using-Seaborn.ipynb Initializes a QuantBook object to interact with the QuantConnect platform. It defines security tickers for line, scatter, and heatmap plots, fetches historical daily closing price data using `qb.History` for the specified date range, and then structures the resulting DataFrame for use in subsequent visualization steps. Requires the QuantConnect environment. ```python qb = QuantBook() # Make symbols for each plot line_tickers = ['SPY'] line_symbols = [qb.Symbol(ticker) for ticker in line_tickers] scatter_tickers = ["BAR", "GLD"] scatter_symbols = [qb.Symbol(ticker) for ticker in scatter_tickers] heatmap_tickers = [ "BAC", # Bank of America Corporation "COF", # Capital One Financial Corporation "C", # Citigroup Inc. "JPM", # J P Morgan Chase & Co "WFC", # Wells Fargo & Company ] heatmap_symbols = [qb.Symbol(ticker) for ticker in heatmap_tickers] # Make history request history = qb.History(line_symbols + scatter_symbols + heatmap_symbols, datetime(2019, 1, 1), datetime(2020, 7, 1), Resolution.Daily).close.unstack(level=0) # Seperate the history for each plot line_history = history[[str(symbol.ID) for symbol in line_symbols]] scatter_history = history[[str(symbol.ID) for symbol in scatter_symbols]] heatmap_history = history[[str(symbol.ID) for symbol in heatmap_symbols]] ``` -------------------------------- ### Initializing Data & Fetching History with QuantConnect Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/08 Long Short-Term Memory.ipynb Imports necessary libraries for data manipulation, scaling, visualization, and deep learning (QuantConnect, Keras, numpy, pandas, matplotlib, sklearn). Initializes the QuantBook environment, adds the SPY security, fetches its historical daily close price data using the QuantConnect API for a specified duration, and splits the data into total, training (first 1260 days), and test sets (remaining). Reshapes the training data into a NumPy array for scaling. ```Python import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.preprocessing import MinMaxScaler # Import keras modules from keras.layers import LSTM from keras.layers import Dense from keras.layers import Dropout from keras.models import Sequential qb = QuantBook() symbol = qb.AddEquity("SPY").Symbol # Fetch history history = qb.History([symbol], 1280, Resolution.Daily) # Fetch price total_price = history.loc[symbol].close training_price = history.loc[symbol].close[:1260] test_price = history.loc[symbol].close[1260:] # Transform price price_array = np.array(training_price).reshape((len(training_price), 1)) ``` -------------------------------- ### Importing Libraries for Model Building Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-Sklearn.ipynb Imports necessary libraries from scikit-learn for building and selecting models, as well as the pickle library for serializing the model to save and load it later. ```python from sklearn.svm import SVR from sklearn.model_selection import GridSearchCV from sklearn.model_selection import train_test_split import pickle ``` -------------------------------- ### Fetching SPY Historical Data using QuantBook (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Plotting-Price-Data-Using-Matplotlib.ipynb Initializes the QuantBook environment and retrieves 360 days of historical daily closing prices for the SPY equity. It then isolates the 'close' price data for SPY into a pandas Series for plotting. ```python qb = QuantBook() spy = qb.AddEquity("SPY") history = qb.History(qb.Securities.Keys, 360, Resolution.Daily) spy_hist = history.loc['SPY']['close'] spy_hist ``` -------------------------------- ### Gathering Historical Data QuantConnect Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-TensorFlow.ipynb Initializes the QuantBook environment and retrieves historical minute-resolution closing price data for the SPY ticker symbol over a specified date range. ```python qb = QuantBook() spy = qb.AddEquity("SPY").Symbol data = qb.History(spy, datetime(2020, 6, 22), datetime(2020, 6, 27), Resolution.Minute).loc[spy].close data ``` -------------------------------- ### Initializing Random Forest and Preparing Data in QuantConnect Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/02 Random Forest Regression.ipynb This snippet initializes the QuantBook environment, defines the universe of US Treasuries ETFs, adds them to QuantBook, fetches historical hourly data, calculates returns, creates a synthetic target variable, splits the data into training and testing sets, and initializes the sklearn RandomForestRegressor model. ```Python # QuantBook Analysis Tool # For more information see [https://www.quantconnect.com/docs/research/overview] from sklearn.ensemble import RandomForestRegressor from sklearn.model_selection import train_test_split qb = QuantBook() qb symbols = {} assets = ["SHY", "TLT", "SHV", "TLH", "EDV", "BIL", "SPTL", "TBT", "TMF", "TMV", "TBF", "VGSH", "VGIT", "VGLT", "SCHO", "SCHR", "SPTS", "GOVT"] for i in range(len(assets)): symbols[assets[i]] = qb.AddEquity(assets[i],Resolution.Minute).Symbol #Copy Paste Region For Backtesting. #========================================== # Set up classifier # Initialize instance of Random Forest Regressor regressor = RandomForestRegressor(n_estimators=100, min_samples_split=5, random_state = 1990) # Fetch history on our universe df = qb.History(qb.Securities.Keys, 500, Resolution.Hour) # Get train/test data returns = df.unstack(level=1).close.transpose().pct_change().dropna() X = returns # use real portfolio value in algo: y = [x for x in qb.portfolioValue][-X.shape[0]:] y = np.random.normal(100000, 5, X.shape[0]) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1990) ``` -------------------------------- ### Training Keras Model on SPY Data Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-Keras.ipynb Prepares the historical SPY data using the `prep_data` function, builds the Keras model using `build_model`, splits the processed data into training and testing sets (80/20 split based on the first 300 samples), and trains the model using the training features and labels for 5 epochs. ```Python X, y = prep_data(spy_hist) model = build_model() # split data into training/testing sets X_train = X[:300] X_test = X[300:] y_train = y[:300] y_test = y[300:] model.fit(X_train, y_train, epochs=5) ``` -------------------------------- ### Fetching Historical Data and Calculating Returns QuantConnect Python Source: https://github.com/quantconnect/research/blob/master/Research2Production/Python/03 Uncorrelated Assets.ipynb Initializes the QuantBook environment and adds a predefined list of equity tickers. It fetches 150 hours of historical data for these securities. The snippet then processes this history to calculate hourly percentage returns, dropping any rows with missing data. Requires the QuantConnect `QuantBook` object. ```python qb = QuantBook() tickers = ["SQQQ", "TQQQ", "TVIX", "VIXY", "SPLV", "SVXY", "UVXY", "EEMV", "EFAV", "USMV"] symbols = [qb.AddEquity(x, Resolution.Minute) for x in tickers] # Fetch history history = qb.History(qb.Securities.Keys, 150, Resolution.Hour) # Get hourly returns returns = history.unstack(level = 1).close.transpose().pct_change().dropna() ``` -------------------------------- ### Building TensorFlow Graph Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-TensorFlow.ipynb Defines the architecture of the neural network using TensorFlow. This includes setting up placeholders for input ('X') and target ('Y'), defining weights and biases with specific initializers, and constructing three hidden layers followed by an output layer using ReLU activation. ```python tf.reset_default_graph() sess = tf.Session() num_factors = X_test.shape[1] num_neurons_1 = 32 num_neurons_2 = 16 num_neurons_3 = 8 X = tf.placeholder(dtype=tf.float32, shape=[None, num_factors], name='X') Y = tf.placeholder(dtype=tf.float32, shape=[None]) # Initializers weight_initializer = tf.variance_scaling_initializer(mode="fan_avg", distribution="uniform", scale=1) bias_initializer = tf.zeros_initializer() # Hidden weights W_hidden_1 = tf.Variable(weight_initializer([num_factors, num_neurons_1])) bias_hidden_1 = tf.Variable(bias_initializer([num_neurons_1])) W_hidden_2 = tf.Variable(weight_initializer([num_neurons_1, num_neurons_2])) bias_hidden_2 = tf.Variable(bias_initializer([num_neurons_2])) W_hidden_3 = tf.Variable(weight_initializer([num_neurons_2, num_neurons_3])) bias_hidden_3 = tf.Variable(bias_initializer([num_neurons_3])) # Output weights W_out = tf.Variable(weight_initializer([num_neurons_3, 1])) bias_out = tf.Variable(bias_initializer([1])) # Hidden layer hidden_1 = tf.nn.relu(tf.add(tf.matmul(X, W_hidden_1), bias_hidden_1)) hidden_2 = tf.nn.relu(tf.add(tf.matmul(hidden_1, W_hidden_2), bias_hidden_2)) hidden_3 = tf.nn.relu(tf.add(tf.matmul(hidden_2, W_hidden_3), bias_hidden_3)) # Output layer output = tf.transpose(tf.add(tf.matmul(hidden_3, W_out), bias_out), name='outer') ``` -------------------------------- ### Fetching GLD and BAR Historical Data and Calculating Returns (Python) Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Plotting-Price-Data-Using-Matplotlib.ipynb Adds GLD and BAR equities to the QuantBook, fetches 300 days of historical daily closing prices for both ETFs, and then calculates the daily percentage change (returns) for each. NaN values resulting from the `pct_change()` calculation are removed. ```python qb.AddEquity('GLD') qb.AddEquity('BAR') gold_etf_hist = qb.History(['GLD', 'BAR'], 300, Resolution.Daily) gld_ret = gold_etf_hist.loc['GLD']['close'].pct_change().dropna() bar_ret = gold_etf_hist.loc['BAR']['close'].pct_change().dropna() ``` -------------------------------- ### Defining Model Testing Function Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-TensorFlow.ipynb Defines a helper Python function `test_model` that takes a TensorFlow session, output tensor, plot title, and input placeholder as arguments. It uses these to generate predictions on the test set and plots them against the actual test data. ```python def test_model(sess, output, title, X): prediction = sess.run(output, feed_dict={X: X_test}) prediction = prediction.reshape(prediction.shape[1], 1) y_test.reset_index(drop=True).plot(figsize=(16, 6), label="Actual") plt.plot(prediction, label="Prediction") plt.title(title) plt.xlabel("Time step") plt.ylabel("SPY Price") plt.legend() plt.show() ``` -------------------------------- ### Helper Functions for ObjectStore Management Python Source: https://github.com/quantconnect/research/blob/master/Documentation/Python/Predicting-Future-Prices-With-TensorFlow.ipynb Provides utility functions to interact with the QuantConnect ObjectStore: `get_ObjectStore_keys` retrieves a list of all keys currently stored, and `clear_ObjectStore` deletes all entries from the ObjectStore. ```python def get_ObjectStore_keys(): return [str(j).split(',')[0][1:] for _, j in enumerate(qb.ObjectStore.GetEnumerator())] def clear_ObjectStore(): for key in get_ObjectStore_keys(): qb.ObjectStore.Delete(key) ```