Tidy Finance

https://www.tidy-finance.org/
Admin
Tidy Finance is an open-source, opinionated approach to reproducible empirical research in financial...
Tokens:250,395
Snippets:1,715
Trust Score:8.6
Update:6 hours ago
Show doc for...
Tokens:
Raw
### Install tidyfinance from PyPI

Source: https://www.tidy-finance.org/chapters/the-tidyfinance-package.html

Installs the released version of the Python package from PyPI.

```Python
pip install tidyfinance
```

--------------------------------

### Install development version of tidyfinance from GitHub (R)

Source: https://www.tidy-finance.org/chapters/the-tidyfinance-package.html

Installs the development version of the R package from GitHub using pak. The commented line shows how to install pak if needed.

```R
# install.packages("pak")
pak::pak("tidy-finance/r-tidyfinance")
```

--------------------------------

### Load Python packages and set polars backend

Source: https://www.tidy-finance.org/chapters/working-with-stock-returns.html

Imports polars and tidyfinance, then sets the tidyfinance backend to polars so download functions return polars data frames. Requires prior installation via pip install polars tidyfinance.

```Python
import polars as pl
import tidyfinance as tf

tf.set_backend("polars")
```

--------------------------------

### Install development version of tidyfinance from GitHub (Python)

Source: https://www.tidy-finance.org/chapters/the-tidyfinance-package.html

Installs the development version of the Python package from GitHub using pip.

```Python
pip install "git+https://github.com/tidy-finance/py-tidyfinance"
```

--------------------------------

### Add credentials to .Renviron

Source: https://www.tidy-finance.org/chapters/setting-up-your-environment.html

Example KEY=value pairs for WRDS credentials in the .Renviron file. Replace with your actual username and password.

```plaintext
WRDS_USER=your_username
WRDS_PASSWORD=your_password
```

--------------------------------

### Python setup for DCF analysis

Source: https://www.tidy-finance.org/chapters/discounted-cash-flow-analysis.html

Imports Python libraries (polars, numpy, pyfixest, fmpapi, plotnine, mizani) and loads environment variables.

```Python
import polars as pl
import numpy as np
import pyfixest as pf

from dotenv import load_dotenv
from fmpapi import fmp_get
from plotnine import *
from mizani.formatters import percent_format, comma_format
from itertools import product

load_dotenv()
```

--------------------------------

### Backtest setup in Python

Source: https://www.tidy-finance.org/chapters/constrained-optimization-and-backtesting.html

Defines the rolling-window parameters (window_length=120, beta=50, gamma=2) and initializes performance tracking dictionaries and previous weights for the three strategies. This is the first step of the backtest workflow.

```Python
window_length = 120
periods = industry_returns.shape[0] - window_length

beta = 50
gamma = 2

performance_values = np.empty((periods, 3))
performance_values[:] = np.nan
performance_values = {
    "MV (TC)": performance_values.copy(),
    "Naive": performance_values.copy(),
    "MV": performance_values.copy(),
}

w_prev_1 = w_prev_2 = w_prev_3 = np.ones(n_industries) / n_industries
```

--------------------------------

### Python usage example for compute_portfolio_weights

Source: https://www.tidy-finance.org/chapters/parametric-portfolio-policies.html

Calls the compute_portfolio_weights function in Python with value weighting and short selling allowed. This example uses the theta and data_portfolios objects.

```python
weights_crsp = compute_portfolio_weights(
    theta,
    data_portfolios,
    value_weighting=True,
    allow_short_selling=True
)
```

--------------------------------

### Backtest setup in R

Source: https://www.tidy-finance.org/chapters/constrained-optimization-and-backtesting.html

Defines the rolling-window parameters (window_length=120, beta=50, gamma=2) and initializes performance tracking matrices and previous weights for the three strategies. This is the first step of the backtest workflow.

```R
window_length <- 120
periods <- nrow(industry_returns) - window_length

beta <- 50
gamma <- 2

performance_values <- matrix(NA,
  nrow = periods,
  ncol = 3
)
colnames(performance_values) <- c("raw_return", "turnover", "net_return")

performance_values <- list(
  "MV (TC)" = performance_values,
  "Naive" = performance_values,
  "MV" = performance_values
)

w_prev_1 <- w_prev_2 <- w_prev_3 <- rep(
  1 / n_industries,
  n_industries
)
```

--------------------------------

### Install tidyfinance from CRAN

Source: https://www.tidy-finance.org/chapters/the-tidyfinance-package.html

Installs the released version of the R package from CRAN.

```R
install.packages("tidyfinance")
```

--------------------------------

### R setup for DCF analysis

Source: https://www.tidy-finance.org/chapters/discounted-cash-flow-analysis.html

Loads required R packages (tidyverse, tidyfinance, scales, fmpapi) and sets a minimal theme for plots.

```R
library(tidyverse)
library(tidyfinance)
library(scales)
library(fmpapi)

theme_set(theme_minimal())
```

--------------------------------

### R usage example for compute_portfolio_weights

Source: https://www.tidy-finance.org/chapters/parametric-portfolio-policies.html

Calls the compute_portfolio_weights function in R with value weighting and short selling allowed. This example uses the theta and data_portfolios objects.

```r
weights_crsp <- compute_portfolio_weights(
  theta,
  data_portfolios,
  value_weighting = TRUE,
  allow_short_selling = TRUE
)
```

--------------------------------

### Load R packages and set theme

Source: https://www.tidy-finance.org/chapters/working-with-stock-returns.html

Loads tidyverse, tidyfinance, and scales, and sets a minimal theme for plots. Requires prior installation via install.packages("tidyfinance").

```R
library(tidyverse)
library(tidyfinance)
library(scales)

theme_set(theme_minimal())
```

--------------------------------

### Restore Python dependencies with uv sync

Source: https://www.tidy-finance.org/chapters/setting-up-your-environment.html

Run this command to install the Python dependencies pinned in uv.lock. If Positron detects pyproject.toml, it may offer to create the virtual environment automatically, in which case this command can be skipped.

```bash
uv sync
```

--------------------------------

### Define date range in Python

Source: https://www.tidy-finance.org/chapters/accessing-and-managing-financial-data.html

Sets the start and end dates as strings for fetching financial data. The range starts in 1960 to cover asset pricing studies from 1962 onward.

```Python
start_date = "1960-01-01"
end_date = "2024-12-31"
```

--------------------------------

### Python: Define p_hacking_setup list with itertools.product

Source: https://www.tidy-finance.org/chapters/size-sorts-and-p-hacking.html

Creates a list of all combinations of portfolio counts, exchanges, weighting, and data frames for the size premium robustness tests. The data list includes the full CRSP data, filtered to exclude the Finance industry, and truncated before and after 1990-06-01.

```Python
n_portfolios = [2, 5, 10]
exchanges = [["NYSE"], ["NYSE", "NASDAQ", "AMEX"]]
value_weighted = [True, False]
data = [
    crsp_monthly,
    crsp_monthly.filter(pl.col("industry") != "Finance"),
    crsp_monthly.filter(pl.col("date") < pl.date(1990, 6, 1)),
    crsp_monthly.filter(pl.col("date") >= pl.date(1990, 6, 1)),
]
p_hacking_setup = list(
    product(n_portfolios, exchanges, value_weighted, data)
)
```

--------------------------------

### Define date range in R

Source: https://www.tidy-finance.org/chapters/accessing-and-managing-financial-data.html

Sets the start and end dates for fetching financial data, using the lubridate ymd function. The range starts in 1960 to cover asset pricing studies from 1962 onward.

```R
start_date <- ymd("1960-01-01")
end_date <- ymd("2024-12-31")
```

--------------------------------

### Plot comparison of beta estimates (Python)

Source: https://www.tidy-finance.org/chapters/beta-estimation.html

Plots the comparison of beta estimates using monthly and daily data for example stocks. The Python code joins the combined beta table with example stocks and uses plotnine with faceting by company.

```Python
beta_comparison = beta.join(examples, how="inner", on="permno")

beta_comparison_figure = (
    ggplot(
        beta_comparison,
        aes(x="date", y="beta", color="return_type", linetype="return_type"),
    )
    + geom_line()
    + facet_wrap("~company", ncol=1)
    + labs(
        x="",
        y="",
        color="",
        linetype="",
        title="Comparison of beta estimates using monthly and daily data",
    )
    + scale_x_date(date_breaks="10 years", date_labels="%Y")
    + theme(figure_size=(6.4, 6.4))
)
beta_comparison_figure.show()
```

--------------------------------

### Plot comparison of beta estimates (R)

Source: https://www.tidy-finance.org/chapters/beta-estimation.html

Plots the comparison of beta estimates using monthly and daily data for example stocks. The R code joins the combined beta table with example stocks and uses ggplot2 with faceting by company.

```R
beta |>
  inner_join(examples, join_by(permno)) |>
  ggplot(aes(
    x = date,
    y = beta,
    color = return_type,
    linetype = return_type
  )) +
  geom_line() +
  facet_wrap(~company, ncol = 1) +
  labs(
    x = NULL,
    y = NULL,
    color = NULL,
    linetype = NULL,
    title = "Comparison of beta estimates using monthly and daily data"
  )
```

--------------------------------

### Set WRDS credentials and connect with tidyfinance in Python

Source: https://www.tidy-finance.org/chapters/wrds-crsp-and-compustat.html

Uses the tidyfinance package to set login credentials and create a WRDS connection.

```Python
tf.set_wrds_credentials()
wrds = tf.get_wrds_connection()
```

--------------------------------

### R: Compute and plot rolling-window CAPM betas

Source: https://www.tidy-finance.org/chapters/beta-estimation.html

Joins CAPM examples with example stock data, filters for the market excess coefficient, and plots monthly beta estimates over time. Uses a five-year rolling window with monthly data and adjusted excess returns from CRSP.

```R
beta_examples <- capm_examples |>
  left_join(examples, join_by(permno)) |>
  filter(coefficient == "mkt_excess")

beta_examples |>
  ggplot(aes(x = date, y = estimate, color = company, linetype = company)) +
  geom_line() +
  labs(
    x = NULL,
    y = NULL,
    color = NULL,
    linetype = NULL,
    title = "Monthly beta estimates for example stocks using 5 years of data"
  )
```

--------------------------------

### Apply roll_capm_estimation to example stocks in R

Source: https://www.tidy-finance.org/chapters/beta-estimation.html

Filters the CRSP monthly data to the example stocks and applies the roll_capm_estimation function. The function groups by permno internally, so no looping is needed. The resulting tibble is printed, showing 3,114 rows with columns permno, date, coefficient, estimate, and t_statistic.

```r
capm_examples <- crsp_monthly |>
  filter(permno %in% examples$permno) |>
  roll_capm_estimation()
capm_examples
```

```r
# A tibble: 3,114 × 5
  permno date       coefficient estimate t_statistic
   <dbl> <date>     <chr>          <dbl>       <dbl>
1  10107 1990-03-01 alpha         0.0417        2.31
2  10107 1990-03-01 mkt_excess    1.40          4.17
3  10107 1990-04-01 alpha         0.0427        2.41
4  10107 1990-04-01 mkt_excess    1.39          4.20
5  10107 1990-05-01 alpha         0.0443        2.53
# ℹ 3,109 more rows
```

--------------------------------

### Compute monthly returns (Python)

Source: https://www.tidy-finance.org/chapters/modern-portfolio-theory.html

Aggregates daily prices to monthly frequency and computes monthly returns as the percentage change from the previous month.

```Python
returns_monthly = (prices_daily
    .sort("date")
    .with_columns(date=pl.col("date").dt.truncate("1mo"))
    .group_by("symbol", "date")
    .agg(adjusted_close=pl.col("adjusted_close").last())
    .sort("symbol", "date")
    .with_columns(ret=pl.col("adjusted_close").pct_change().over("symbol"))
    .drop_nulls("ret")
    .select("symbol", "date", "ret")
)
```

--------------------------------

### Run SMB regression and display summary (Python)

Source: https://www.tidy-finance.org/chapters/replicating-fama-and-french-factors.html

Runs a linear regression of the original SMB factor on the replicated SMB factor using pf.feols() and prints the summary. The output shows a non-significant intercept, a coefficient of 0.979, and an R-squared of 0.987, indicating a successful replication.

```Python
model_smb = pf.feols(
    fml="smb ~ smb_replicated",
    data=test
)
model_smb.summary()
```

```Python output
###

Estimation:  OLS
Dep. var.: smb
sample: None = all
Inference:  iid
Observations:  762

| Coefficient    |   Estimate |   Std. Error |   t value |   Pr(>|t|) |   2.5% |   97.5% |
|:---------------|-----------:|-------------:|----------:|-----------:|-------:|--------:|
| Intercept      |     -0.000 |        0.000 |    -1.368 |      0.172 | -0.000 |   0.000 |
| smb_replicated |      0.979 |        0.004 |   238.380 |      0.000 |  0.971 |   0.987 |
---
RMSE: 0.003 R2: 0.987 
```

--------------------------------

### Apply roll_capm_estimation to example stocks in Python (Polars)

Source: https://www.tidy-finance.org/chapters/beta-estimation.html

Filters the CRSP monthly data to the example stocks and applies the roll_capm_estimation function. The function groups by permno internally, so no looping is needed. The resulting DataFrame is printed, showing 3,114 rows with columns permno, date, coefficient, estimate, and t_statistic.

```python
capm_examples = roll_capm_estimation(
    crsp_monthly.filter(pl.col("permno").is_in(examples["permno"]))
)
capm_examples
```

```python
shape: (3_114, 5)

| permno | date | coefficient | estimate | t_statistic |
| --- | --- | --- | --- | --- |
| f64 | date | str | f64 | f64 |
| --- | --- | --- | --- | --- |
| 10107.0 | 1990-03-01 | "alpha" | 0.041709 | 2.309559 |
| 10107.0 | 1990-03-01 | "mkt_excess" | 1.397341 | 4.173565 |
| 10107.0 | 1990-04-01 | "alpha" | 0.042693 | 2.413745 |
| 10107.0 | 1990-04-01 | "mkt_excess" | 1.385035 | 4.197367 |
| 10107.0 | 1990-05-01 | "alpha" | 0.044259 | 2.533768 |
| … | … | … | … | … |
| 93436.0 | 2024-10-01 | "mkt_excess" | 2.380626 | 5.488947 |
| 93436.0 | 2024-11-01 | "alpha" | 0.038041 | 1.593754 |
| 93436.0 | 2024-11-01 | "mkt_excess" | 2.451092 | 5.645404 |
| 93436.0 | 2024-12-01 | "alpha" | 0.039498 | 1.654254 |
| 93436.0 | 2024-12-01 | "mkt_excess" | 2.385498 | 5.496126 |
```

--------------------------------

### Python: Compute and plot rolling-window CAPM betas

Source: https://www.tidy-finance.org/chapters/beta-estimation.html

Joins CAPM examples with example stock data, filters for the market excess coefficient, and plots monthly beta estimates over time. Uses a five-year rolling window with monthly data and adjusted excess returns from CRSP. The plot is displayed with .show().

```Python
beta_examples = (capm_examples
    .join(examples, how="left", on="permno")
    .filter(pl.col("coefficient") == "mkt_excess")
)

beta_figure = (
    ggplot(
        beta_examples,
        aes(x="date", y="estimate", color="company", linetype="company"),
    )
    + geom_line()
    + labs(
        x="",
        y="",
        color="",
        linetype="",
        title="Monthly beta estimates for example stocks using 5 years of data",
    )
    + scale_x_date(date_breaks="5 year", date_labels="%Y")
)
beta_figure.show()
```

--------------------------------

### Download pseudo crsp_daily table (Python)

Source: https://www.tidy-finance.org/chapters/wrds-dummy-data.html

Downloads the daily CRSP pseudo data and writes it to data/crsp_daily.parquet. The daily returns are used for estimating betas from daily data.

```Python
crsp_daily = tf.download_data(
    domain="Pseudo Data",
    dataset="crsp_daily",
    start_date=start_date,
    end_date=end_date,
    n_assets=n_assets,
    seed=seed
)

crsp_daily.write_parquet("data/crsp_daily.parquet")
```

--------------------------------

### R: Set up nloptr options and solve minimum variance portfolio

Source: https://www.tidy-finance.org/chapters/constrained-optimization-and-backtesting.html

Defines the objective function, gradient, equality constraint, and Jacobian for the minimum variance portfolio, then calls nloptr with the SLSQP algorithm. The equality constraint requires weights to sum to one. The output is verified against the analytical solution using near(), which returns TRUE.

```R
w_initial <- rep(1 / n_industries, n_industries)

objective_mvp <- function(w) {
  0.5 * t(w) %*% Sigma %*% w
}

gradient_mvp <- function(w) {
  Sigma %*% w
}

equality_constraint <- function(w) {
  sum(w) - 1
}

jacobian_equality <- function(w) {
  rep(1, n_industries)
}

options <- list(
  "xtol_rel"=1e-20,
  "algorithm" = "NLOPT_LD_SLSQP",
  "maxeval" = 10000
)

w_mvp_numerical <- nloptr(
  x0 = w_initial,
  eval_f = objective_mvp,
  eval_grad_f = gradient_mvp,
  eval_g_eq = equality_constraint,
  eval_jac_g_eq = jacobian_equality,
  opts = options
)

all(near(w_mvp, w_mvp_numerical$solution))
```

```R
[1] TRUE
```

--------------------------------

### Python: compute_efficient_weight function and example call

Source: https://www.tidy-finance.org/chapters/constrained-optimization-and-backtesting.html

Defines compute_efficient_weight in Python, which computes efficient portfolio weights with optional transaction costs. The function uses numpy operations and returns the optimal weight vector. The example call uses sigma and mu and creates a Polars DataFrame with the weights rounded to 3 decimals.

```Python
def compute_efficient_weight(
    sigma, mu, gamma=2, beta=0, w_prev=np.ones(sigma.shape[1]) / sigma.shape[1]
):
    """Compute efficient portfolio weights."""

    n = sigma.shape[1]
    iota = np.ones(n)
    sigma_processed = sigma + (beta / gamma) * np.eye(n)
    mu_processed = mu + beta * w_prev

    sigma_inverse = np.linalg.inv(sigma_processed)

    w_mvp = sigma_inverse @ iota
    w_mvp = w_mvp / np.sum(w_mvp)
    w_opt = (
        w_mvp
        + (1 / gamma)
        * (sigma_inverse - np.outer(w_mvp, iota) @ sigma_inverse)
        @ mu_processed
    )

    return w_opt


w_efficient = compute_efficient_weight(sigma, mu)

weights_efficient = pl.DataFrame(
    {
        "Industry": industry_returns.columns,
        "Efficient portfolio": w_efficient,
    }
)
weights_efficient.with_columns(pl.col(pl.Float64).round(3))
```

```Python output
shape: (10, 2)

| Industry | Efficient portfolio |
| --- | --- |
| str | f64 |
| --- | --- |
| "nodur" | 1.185 |
| "durbl" | 0.237 |
| "manuf" | -1.64 |
| "enrgy" | 0.614 |
| "hitec" | 0.449 |
| "telcm" | -0.437 |
| "shops" | 0.754 |
| "hlth" | 0.358 |
| "utils" | -0.047 |
| "other" | -0.474 |
```

--------------------------------

### R: compute_efficient_weight function and example call

Source: https://www.tidy-finance.org/chapters/constrained-optimization-and-backtesting.html

Defines compute_efficient_weight in R, which computes efficient portfolio weights with optional transaction costs. The function uses the inverse of a modified covariance matrix and returns the optimal weight vector. The example call uses Sigma and mu and prints the weights rounded to 3 decimals.

```R
compute_efficient_weight <- function(Sigma,
                                     mu,
                                     gamma = 2,
                                     beta = 0, # transaction costs
                                     w_prev = rep(
                                       1 / ncol(Sigma),
                                       ncol(Sigma)
                                     )) {
  iota <- rep(1, ncol(Sigma))
  Sigma_processed <- Sigma + beta / gamma * diag(ncol(Sigma))
  mu_processed <- mu + beta * w_prev

  Sigma_inverse <- solve(Sigma_processed)

  w_mvp <- Sigma_inverse %*% iota
  w_mvp <- as.vector(w_mvp / sum(w_mvp))
  w_opt <- w_mvp + 1 / gamma *
    (Sigma_inverse - w_mvp %*% t(iota) %*% Sigma_inverse) %*%
      mu_processed

  return(as.vector(w_opt))
}

w_efficient <- compute_efficient_weight(Sigma, mu)
round(w_efficient, 3)
```

```R output
 [1]  1.185  0.237 -1.640  0.614  0.449 -0.437  0.754  0.358 -0.047
[10] -0.474
```

--------------------------------

### Load Python packages (polars, numpy, pyfixest, scipy)

Source: https://www.tidy-finance.org/chapters/parametric-portfolio-policies.html

Imports polars, numpy, pyfixest, and scipy.optimize for data manipulation, numerical operations, fixed-effects estimation, and optimization. Also imports itertools for product and starmap.

```Python
import polars as pl
import numpy as np
import pyfixest as pf

from itertools import product, starmap
from scipy.optimize import minimize
```

--------------------------------

### Python: Import required packages and set backend

Source: https://www.tidy-finance.org/chapters/capital-asset-pricing-model.html

Imports polars, numpy, tidyfinance, pyfixest, plotnine, mizani formatters, and adjustText. Sets the tidyfinance backend to 'polars' and the plot theme to theme_minimal().

```Python
import polars as pl
import numpy as np
import tidyfinance as tf
import pyfixest as pf

from plotnine import *
from mizani.formatters import percent_format
from adjustText import adjust_text

tf.set_backend("polars")
theme_set(theme_minimal())
```

--------------------------------

### Python: compute_dcf function and example output

Source: https://www.tidy-finance.org/chapters/discounted-cash-flow-analysis.html

Defines compute_dcf in Python, which calculates the total DCF value using the WACC and growth rate. It calls compute_terminal_value and returns the sum of present values of free cash flows and terminal value. The example call with wacc and 0.04 growth rate prints 6550.0 (in billions).

```Python
def compute_dcf(wacc, growth_rate):
    free_cash_flow = dcf_data["fcf"].to_numpy()
    last_fcf = free_cash_flow[-1]
    terminal_value = compute_terminal_value(last_fcf, growth_rate, wacc)

    years = len(free_cash_flow)
    present_value_fcf = free_cash_flow / ((1 + wacc) ** np.arange(1, years + 1))
    present_value_tv = terminal_value / ((1 + wacc) ** years)

    return present_value_fcf.sum() + present_value_tv

dcf_value = compute_dcf(wacc, 0.04)

print(np.round(dcf_value / 1e9))
```

```Python output
6550.0
```

--------------------------------

### R: compute_dcf function and example output

Source: https://www.tidy-finance.org/chapters/discounted-cash-flow-analysis.html

Defines compute_dcf in R, which calculates the total DCF value using the WACC and growth rate. It calls compute_terminal_value and returns the sum of present values of free cash flows and terminal value. The example call with wacc and 0.04 growth rate outputs 6550 (in billions).

```R
compute_dcf <- function(wacc, growth_rate) {
  free_cash_flow <- dcf_data$fcf
  last_fcf <- tail(free_cash_flow, 1)
  terminal_value <- compute_terminal_value(last_fcf, growth_rate, wacc)

  years <- length(free_cash_flow)
  present_value_fcf <- free_cash_flow / (1 + wacc)^(1:years)
  present_value_tv <- terminal_value / (1 + wacc)^years
  total_dcf_value <- sum(present_value_fcf) + present_value_tv
  total_dcf_value
}

compute_dcf(wacc, 0.04) / 1e9
```

```R output
[1] 6550
```

--------------------------------

### Download and prepare monthly returns (Python)

Source: https://www.tidy-finance.org/chapters/capital-asset-pricing-model.html

Downloads Dow Jones constituents and daily prices, filters to the most recent period, and computes monthly returns. Uses the tidyfinance and polars libraries.

```Python
symbols = tf.download_data(
    domain="Index Constituents", index="Dow Jones Industrial Average"
)

prices_daily = tf.download_data(
    domain="Stock Prices",
    symbols=symbols["symbol"].to_list(),
    start_date="2000-01-01",
    end_date="2024-12-31",
).select("symbol", "date", "adjusted_close")

prices_daily = prices_daily.with_columns(
    counts=pl.col("adjusted_close").drop_nulls().len().over("symbol")
).filter(pl.col("counts") == pl.col("counts").max())

returns_monthly = (
    prices_daily.sort("symbol", "date")
    .with_columns(month=pl.col("date").dt.truncate("1mo"))
    .group_by("symbol", "month", maintain_order=True)
    .agg(adjusted_close=pl.col("adjusted_close").last())
    .rename({"month": "date"})
    .sort("symbol", "date")
    .with_columns(ret=pl.col("adjusted_close").pct_change().over("symbol"))
    .drop_nulls("ret")
    .select("symbol", "date", "ret")
)
```

--------------------------------

### Load and prepare data for portfolio sorts (R)

Source: https://www.tidy-finance.org/chapters/univariate-portfolio-sorts.html

Loads CRSP monthly returns, Fama-French factors, and precomputed betas from parquet files. Selects relevant columns for portfolio construction.

```R
crsp_monthly <- read_parquet("data/crsp_monthly.parquet") |>
  select(permno, date, ret_excess, mktcap_lag)

factors_ff3_monthly <- read_parquet("data/factors_ff3_monthly.parquet") |>
  select(date, mkt_excess)

beta <- read_parquet("data/beta.parquet") |>
  filter(return_type == "monthly") |>
  select(permno, date, beta)
```

--------------------------------

### Set date range in Python for WRDS data

Source: https://www.tidy-finance.org/chapters/wrds-crsp-and-compustat.html

Defines start and end dates as strings in the format expected by the WRDS database.

```Python
start_date = "01/01/1960"
end_date = "12/31/2024"
```

--------------------------------

### Load .env file in Python

Source: https://www.tidy-finance.org/chapters/setting-up-your-environment.html

Use python-dotenv to load environment variables from the .env file at the start of your script, then access them with os.getenv().

```Python
import os
from dotenv import load_dotenv

load_dotenv()
wrds_user = os.getenv("WRDS_USER")
```

--------------------------------

### Load required packages (Python)

Source: https://www.tidy-finance.org/chapters/wrds-dummy-data.html

Imports polars, numpy, tidyfinance, string, and os, and sets the backend to polars for pseudo data generation in Python.

```Python
import polars as pl
import numpy as np
import tidyfinance as tf
import string
import os

tf.set_backend("polars")
```

--------------------------------

### read_csv() literal data warning (R)

Source: https://www.tidy-finance.org/chapters/accessing-and-managing-financial-data.html

Shows the warning that read_csv() should use I() for literal data as of readr 2.2.0, with bad and good examples.

```R
Warning: The `file` argument of `read_csv()` should use `I()` for literal
data as of readr 2.2.0.
  
  # Bad (for example):
  read_csv("x,y\n1,2")
  
  # Good:
  read_csv(I("x,y\n1,2"))
```

--------------------------------

### Load and prepare data for portfolio sorts (Python)

Source: https://www.tidy-finance.org/chapters/univariate-portfolio-sorts.html

Loads CRSP monthly returns, Fama-French factors, and precomputed betas from parquet files using Polars. Selects relevant columns for portfolio construction.

```Python
crsp_monthly = (
    pl.read_parquet("data/crsp_monthly.parquet")
    .select("permno", "date", "ret_excess", "mktcap_lag")
)

factors_ff3_monthly = (
    pl.read_parquet("data/factors_ff3_monthly.parquet")
    .select("date", "mkt_excess")
)

beta = (
    pl.read_parquet("data/beta.parquet")
    .filter(pl.col("return_type") == "monthly")
    .select("permno", "date", "beta")
)
```

--------------------------------

### Python: calculate_returns and compute_portfolio_returns functions

Source: https://www.tidy-finance.org/chapters/size-sorts-and-p-hacking.html

Implements value-weighted and equal-weighted returns via calculate_returns, and computes the size premium with compute_portfolio_returns. The function takes a logical value_weighted argument and uses assign_portfolio for breakpoints.

```Python
def calculate_returns(value_weighted):
    """Build the (value-weighted) return expression."""

    if value_weighted:
      return (
          (pl.col("ret_excess") * pl.col("mktcap_lag")).sum()
          / pl.col("mktcap_lag").sum()
      )
    else:
      return pl.col("ret_excess").mean()

def compute_portfolio_returns(n_portfolios=10,
                              exchanges=["NYSE", "NASDAQ", "AMEX"],
                              value_weighted=True,
                              data=crsp_monthly):
    """Compute (value-weighted) portfolio returns."""

    data_assigned = pl.concat([
        group.with_columns(
            portfolio=assign_portfolio(
                group, exchanges, "mktcap_lag", n_portfolios
            )
        )
        for _, group in data.group_by("date")
    ])

    portfolio_returns = (data_assigned
      .group_by("portfolio", "date")
      .agg(ret=calculate_returns(value_weighted))
    )

    size_premia = (portfolio_returns
      .group_by("date")
      .agg(
        size_premium=(
          pl.col("ret").filter(pl.col("portfolio") == pl.col("portfolio").min())
          - pl.col("ret").filter(pl.col("portfolio") == pl.col("portfolio").max())
        ).first()
      )
    )

    return size_premia["size_premium"].mean()
```

--------------------------------

### Python: Load environment variables and create SQLAlchemy engine

Source: https://www.tidy-finance.org/chapters/trace-and-fisd.html

Loads environment variables using python-dotenv and creates a SQLAlchemy engine to connect to WRDS. Requires the WRDS_USER and WRDS_PASSWORD environment variables to be set.

```Python
import os
from dotenv import load_dotenv
from sqlalchemy import create_engine, URL

load_dotenv()
```

```Python
True
```

```Python
connection_url = URL.create(
    drivername="postgresql+psycopg2",
    username=os.getenv("WRDS_USER"),
    password=os.getenv("WRDS_PASSWORD"),
    host="wrds-pgdata.wharton.upenn.edu",
    port=9737,
    database="wrds",
)

wrds = create_engine(connection_url, pool_pre_ping=True)
```

--------------------------------

### Create WRDS connection URL and engine in Python

Source: https://www.tidy-finance.org/chapters/wrds-crsp-and-compustat.html

Builds a SQLAlchemy connection URL for WRDS and creates an engine with pool_pre_ping. Uses credentials from environment variables.

```Python
connection_url = URL.create(
    drivername="postgresql+psycopg2",
    username=os.getenv("WRDS_USER"),
    password=os.getenv("WRDS_PASSWORD"),
    host="wrds-pgdata.wharton.upenn.edu",
    port=9737,
    database="wrds",
)

wrds = create_engine(connection_url, pool_pre_ping=True)
```

--------------------------------

### Load environment variables for WRDS connection in Python

Source: https://www.tidy-finance.org/chapters/wrds-crsp-and-compustat.html

Loads environment variables from a .env file using dotenv, required for the subsequent connection setup.

```Python
import os
from dotenv import load_dotenv
from sqlalchemy import create_engine, URL

load_dotenv()
```
Tokens:
Raw