Macrosynergy

https://docs.macrosynergy.com/stable/
Admin
The Macrosynergy package provides Python classes and functions for analyzing and transforming...
Tokens:810,098
Snippets:4,734
Trust Score:8.8
Update:4 months ago
Show doc for...
Tokens:
Raw
### Get Products cURL Example

Source: https://docs.macrosynergy.com/stable/macrosynergy.download.fusion_interface.html

Demonstrates how to retrieve the list of available products using a cURL command. Requires an Authorization header.

```bash
curl -X GET "https://fusion.jpmorgan.com/api/v1/catalogs/common/products" \
    -H "Authorization: Bearer <ACCESS_TOKEN>"
```

--------------------------------

### Example Dataset Creation and Preparation

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/learning/splitters/walk_forward_splitters.html

Generates a sample quarterly dataframe (qdf) and preprocesses it by reducing data based on blacklisted periods and pivoting it for use in splitting strategies. This setup is crucial before applying any splitting techniques.

```python
from macrosynergy.management.simulate import make_qdf
import macrosynergy.management as msm

# Example dataset
cids = ["AUD", "CAD", "GBP", "USD"]
xcats = ["XR", "CRY", "GROWTH", "INFL"]
cols = ["earliest", "latest", "mean_add", "sd_mult"]

df_cids = pd.DataFrame(
    index=cids, columns=cols
)
df_cids.loc["AUD"] = ["2002-01-01", "2020-12-31", 0, 1]
df_cids.loc["CAD"] = ["2003-01-01", "2020-12-31", 0, 1]
df_cids.loc["GBP"] = ["2000-01-01", "2020-12-31", 0, 1]
df_cids.loc["USD"] = ["2000-01-01", "2020-12-31", 0, 1]

df_xcats = pd.DataFrame(index=xcats, columns=cols)
df_xcats.loc["XR"] = ["2000-01-01", "2020-12-31", 0.1, 1, 0, 0.3]
df_xcats.loc["CRY"] = ["2000-01-01", "2020-12-31", 1, 2, 0.95, 1]
df_xcats.loc["GROWTH"] = ["2001-01-01", "2020-12-31", 1, 2, 0.9, 1]
df_xcats.loc["INFL"] = ["2000-01-01", "2020-12-31", 1, 2, 0.8, 0.5]

dfd = make_qdf(df_cids, df_xcats, back_ar=0.75)
dfd["grading"] = np.ones(dfd.shape[0])
black = {"GBP": ["2009-01-01", "2012-06-30"], "CAD": ["2018-01-01", "2100-01-01"]}
dfd = msm.reduce_df(df=dfd, cids=cids, xcats=xcats, blacklist=black)

dfd = dfd.pivot(index=["cid", "real_date"], columns="xcat", values="value")
X = dfd.drop(columns=["XR"])
y = dfd["XR"]
```

--------------------------------

### Example: Unbalanced Panel Data Setup

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/learning/random_effects.html

Demonstrates the creation of unbalanced panel data using `make_qdf` for a random effects model. Includes defining cross-sectional IDs, cross-asset categories, and column configurations.

```python
if __name__ == "__main__":

    np.random.seed(1)

    cids = ["AUD", "CAD", "GBP", "USD"]
    xcats = ["XR", "CRY", "GROWTH", "INFL"]
    cols = ["earliest", "latest", "mean_add", "sd_mult", "ar_coef", "back_coef"]

    """Example: Unbalanced panel """

    df_cids2 = pd.DataFrame(
        index=cids, columns=["earliest", "latest", "mean_add", "sd_mult"]
    )
    df_cids2.loc["AUD"] = ["2012-01-01", "2020-12-31", 0, 1]
    df_cids2.loc["CAD"] = ["2013-01-01", "2020-12-31", 0, 1]
    df_cids2.loc["GBP"] = ["2010-01-01", "2020-12-31", 0, 1]
    df_cids2.loc["USD"] = ["2010-01-01", "2020-12-31", 0, 1]

    df_xcats2 = pd.DataFrame(index=xcats, columns=cols)
    df_xcats2.loc["XR"] = ["2010-01-01", "2020-12-31", 0.1, 1, 0, 0.3]
    df_xcats2.loc["CRY"] = ["2010-01-01", "2020-12-31", 1, 2, 0.95, 1]
    df_xcats2.loc["GROWTH"] = ["2010-01-01", "2020-12-31", 1, 2, 0.9, 1]
    df_xcats2.loc["INFL"] = ["2010-01-01", "2020-12-31", 1, 2, 0.8, 0.5]

    dfd2 = make_qdf(df_cids2, df_xcats2, back_ar=0.75)
    dfd2["grading"] = np.ones(dfd2.shape[0])
    black = {"GBP": ["2009-01-01", "2012-06-30"], "CAD": ["2018-01-01", "2100-01-01"]}
```

--------------------------------

### Example DataFrame Creation and Table Visualization

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/visuals/table.html

Demonstrates how to create a Pandas DataFrame and then visualize it using the view_table function. This is a basic setup for displaying tabular data.

```python
if __name__ == "__main__":
    data = {
        "Col1": [1, 2, 3, 4],
        "Col2": [5, 6, 7, 8],
        "Col3": [9, 10, 11, 12],
        "Col4": [13, 14, 15, 16],
    }
    row_labels = ["Row1", "Row2", "Row3", "Row4"]

    df = pd.DataFrame(data, index=row_labels)

    view_table(df, title="Table")
```

--------------------------------

### Example Usage with make_qdf and LinearRegression

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/learning/forecasting/meta_estimators/weighted_predictors.html

Demonstrates how to use the TimeWeightedWrapper with a LinearRegression model. This example sets up sample data using make_qdf, defines training and testing sets, and applies the wrapper.

```python
import macrosynergy.management as msm
from macrosynergy.management.simulate import make_qdf
import pandas as pd
import numpy as np

from sklearn.linear_model import LinearRegression


cids = ["AUD", "CAD", "GBP", "USD"]
xcats = ["XR", "CRY", "GROWTH", "INFL"]
cols = ["earliest", "latest", "mean_add", "sd_mult", "ar_coef", "back_coef"]

"""Example: Unbalanced panel """

df_cids = pd.DataFrame(
    index=cids, columns=["earliest", "latest", "mean_add", "sd_mult"]
)
df_cids.loc["AUD"] = ["2002-01-01", "2020-12-31", 0, 1]
df_cids.loc["CAD"] = ["2003-01-01", "2020-12-31", 0, 1]
df_cids.loc["GBP"] = ["2000-01-01", "2020-12-31", 0, 1]
df_cids.loc["USD"] = ["2000-01-01", "2020-12-31", 0, 1]

df_xcats = pd.DataFrame(index=xcats, columns=cols)
df_xcats.loc["XR"] = ["2000-01-01", "2020-12-31", 0.1, 1, 0, 0.3]
df_xcats.loc["CRY"] = ["2000-01-01", "2020-12-31", 1, 2, 0.95, 1]
df_xcats.loc["GROWTH"] = ["2000-01-01", "2020-12-31", 1, 2, 0.9, 1]
df_xcats.loc["INFL"] = ["2000-01-01", "2020-12-31", -0.1, 2, 0.8, 0.3]

dfd = make_qdf(df_cids, df_xcats, back_ar=0.75)
dfd["grading"] = np.ones(dfd.shape[0])
black = {
    "GBP": (
        pd.Timestamp(year=2009, month=1, day=1),
        pd.Timestamp(year=2012, month=6, day=30),
    ),
    "CAD": (
        pd.Timestamp(year=2015, month=1, day=1),
        pd.Timestamp(year=2100, month=1, day=1),
    ),
}

train = msm.categories_df(
    df=dfd, xcats=xcats, cids=cids, val="value", blacklist=black, freq="M", lag=1
).dropna()

# Training set
X_train = train.drop(columns=["XR"])
y_train = np.sign(train["XR"])

# Instantiate the wrapper with a LinearRegression model and a half-life of 1 year
model = LinearRegression()
wrapped_model = TimeWeightedWrapper(model=model, half_life=1.0)

# Fit the wrapped model
wrapped_model.fit(X_train, y_train)

# Predict using the wrapped model
predictions = wrapped_model.predict(X_train)
print(predictions)

```

--------------------------------

### Example: Unbalanced Panel Data Setup

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/learning/forecasting/linear_model/global_local.html

Sets up an unbalanced panel DataFrame for testing purposes, defining entity-specific date ranges and parameters.

```python
import pandas as pd

cids = ["AUD", "CAD", "GBP", "USD"]
cncats = ["XR", "CRY", "GROWTH", "INFL"]
ncols = ["earliest", "latest", "mean_add", "sd_mult", "ar_coef", "back_coef"]

"""Example: Unbalanced panel """

df_cids = pd.DataFrame(
    index=cids, columns=["earliest", "latest", "mean_add", "sd_mult"]
)
df_cids.loc["AUD"] = ["2002-01-01", "2020-12-31", 0, 1]
df_cids.loc["CAD"] = ["2003-01-01", "2020-12-31", 0, 1]
df_cids.loc["GBP"] = ["2000-01-01", "2020-12-31", 0, 1]
df_cids.loc["USD"] = ["2000-01-01", "2020-12-31", 0, 1]

df_xcats = pd.DataFrame(index=xcats, columns=cols)
```

--------------------------------

### Get Product Details cURL Example

Source: https://docs.macrosynergy.com/stable/macrosynergy.download.fusion_interface.html

Demonstrates how to retrieve details for a specific product using its ID via cURL. Requires an Authorization header.

```bash
curl -X GET "https://fusion.jpmorgan.com/api/v1/catalogs/common/products/{product_id}" \
    -H "Authorization: Bearer <ACCESS_TOKEN>"
```

--------------------------------

### Example Usage of make_blacklist

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/panel/make_blacklist.html

Demonstrates the creation of a quantamental DataFrame and its subsequent use with the `make_blacklist` function. This example illustrates how to set up input data and call the function to generate a blacklist dictionary.

```python
if __name__ == "__main__":
    cids = ["AUD", "GBP", "CAD", "USD"]
    cols = ["earliest", "latest", "mean_add", "sd_mult"]
    df_cid1 = pd.DataFrame(index=cids, columns=cols)

    df_cid1.loc["AUD"] = ["2010-01-01", "2020-12-31", 0, 1]
    df_cid1.loc["GBP"] = ["2011-01-01", "2020-11-30", 0, 1]
    df_cid1.loc["CAD"] = ["2011-01-01", "2021-11-30", 0, 1]
    df_cid1.loc["USD"] = ["2011-01-01", "2020-12-30", 0, 1]

    cols = ["earliest", "latest", "mean_add", "sd_mult", "ar_coef", "back_coef"]
    df_xcat1 = pd.DataFrame(index=["FXXR_NSA", "FXCRY_NSA"], columns=cols)
    df_xcat1.loc["FXXR_NSA"] = ["2010-01-01", "2020-12-31", 0, 1, 0, 0.3]
    df_xcat1.loc["FXCRY_NSA"] = ["2010-01-01", "2020-12-31", 0, 1, 0, 0.3]
    df1 = make_qdf(df_cid1, df_xcat1, back_ar=0.05)

    df_xcat2 = pd.DataFrame(index=["FXNONTRADE_NSA"], columns=["earliest", "latest"])
    df_xcat2.loc["FXNONTRADE_NSA"] = ["2010-01-01", "2021-11-30"]
    black = {

```

--------------------------------

### Example Usage of view_performance Function

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/visuals/performance.html

Demonstrates various ways to call the `view_performance` function with different parameters to generate performance visualizations. Includes examples for comparing across cids, xcats, tickers, filtering metrics, and sorting by Sharpe ratio.

```python
if __name__ == "__main__":
    from macrosynergy.management.simulate import make_qdf

    # Create sample data
    cids = ["AUD", "CAD", "GBP", "USD"]
    xcats = ["FXXR_NSA", "EQXR_NSA"]

    df_cids = pd.DataFrame(
        index=cids, columns=["earliest", "latest", "mean_add", "sd_mult"]
    )
    df_cids.loc["AUD"] = ["2010-01-01", "2020-12-31", 0.5, 1.0]
    df_cids.loc["CAD"] = ["2010-01-01", "2020-12-31", 0.3, 1.2]
    df_cids.loc["GBP"] = ["2010-01-01", "2020-12-31", 0.4, 1.1]
    df_cids.loc["USD"] = ["2010-01-01", "2020-12-31", 0.2, 0.9]

    df_xcats = pd.DataFrame(
        index=xcats,
        columns=["earliest", "latest", "mean_add", "sd_mult", "ar_coef", "back_coef"],
    )
    df_xcats.loc["FXXR_NSA"] = ["2010-01-01", "2020-12-31", 0.1, 1, 0, 0.3]
    df_xcats.loc["EQXR_NSA"] = ["2010-01-01", "2020-12-31", 0.3, 1.5, 0, 0.3]

    dfd = make_qdf(df_cids, df_xcats, back_ar=0.75)

    # Test 1: Compare across cids for single xcat
    print("Test 1: Compare across cids")
    view_performance(dfd, xcats=["FXXR_NSA"], cids=["AUD", "CAD", "GBP", "USD"])

    # Test 2: Compare across xcats for single cid
    print("\nTest 2: Compare across xcats")
    view_performance(dfd, xcats=["FXXR_NSA", "EQXR_NSA"], cids=["ALL"])

    # Test 3: Compare specific tickers with benchmark
    print("\nTest 3: Compare tickers with benchmark")
    view_performance(
        dfd,
        tickers=["AUD_FXXR_NSA", "GBP_FXXR_NSA", "USD_FXXR_NSA", "USD_EQXR_NSA"],
        bms="USD_EQXR_NSA"
    )
    
    # Test 4: Filter metrics
    print("\nTest 4: Filter specific metrics")
    view_performance(
        dfd,
        tickers=["AUD_FXXR_NSA", "GBP_FXXR_NSA", "USD_FXXR_NSA", "USD_EQXR_NSA"],
        bms="USD_EQXR_NSA",
        metrics=["USD_EQXR_NSA correl"]
    )

    # Test 5: Sort by Sharpe ratio
    print("\nTest 5: Sort by Sharpe ratio")
    view_performance(
        dfd,
        xcats=["FXXR_NSA"],
        cids=["AUD", "CAD", "GBP", "USD"],
        sort_by="Sharpe ratio"
    )
```

--------------------------------

### Example Usage: Proxy PnL Calculation with Transaction Object

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/pnl/proxy_pnl_calc.html

Demonstrates how to use the proxy_pnl_calc function with a downloaded transaction costs object. This example sets up necessary data and parameters to calculate PnL and costs.

```python
if __name__ == "__main__":
    import os

    import pickle

    cids_dmca = ["AUD", "CAD", "CHF", "EUR", "GBP", "JPY", "NOK", "NZD", "SEK", "USD"]
    cids_dmec = ["DEM", "ESP", "FRF", "ITL"]
    cids_nofx: List[str] = ["USD", "EUR", "CNY", "SGD"]
    cids_dmfx: List[str] = list(set(cids_dmca) - set(cids_nofx))

    if not os.path.exists("data/txn.obj.pkl"):
        with open("data/txn.obj.pkl", "wb") as f:
            pickle.dump(TransactionCosts.download(), f)

    with open("data/txn.obj.pkl", "rb") as f:
        tx = pickle.load(f)

    dfx = pd.read_pickle("data/dfxn.pkl")

    spos = "STRAT_POS"
    rstring = "XR_NSA"
    df_all = proxy_pnl_calc(
        df=dfx,
        spos=spos,
        rstring=rstring,
        start="2001-01-01",
        end="2020-01-01",
        transaction_costs_object=tx,
        portfolio_name="GLB",
        pnl_name="PNL",
        tc_name="TCOST",
        return_pnl_excl_costs=True,
        return_costs=True,
        # concat_dfs=True,
    )

    # Example: dict-based transaction costs
    pos_tickers = [
        t for t in QuantamentalDataFrame(dfx).list_tickers() if t.endswith(f"{spos}")
    ]
    traded_fids = sorted(set(_replace_strs(pos_tickers, f"{spos}")))
    cost_date = tx.change_index[0]
    cost_dict = {}
    for fid in traded_fids:
        row = tx.get_costs(fid=fid, real_date=cost_date)
        if row is None:
            raise ValueError(f"Missing transaction costs for {fid}")
        cost_dict[fid] = {
            "median_cost": row[f"{fid}BIDOFFER_MEDIAN"],
            "median_size": row[f"{fid}SIZE_MEDIAN"],
            "pct90_cost": row[f"{fid}BIDOFFER_90PCTL"],
            "pct90_size": row[f"{fid}SIZE_90PCTL"],
        }

    df_all_dict = proxy_pnl_calc(
        df=dfx,
        spos=spos,
        rstring=rstring,
        start="2001-01-01",
        end="2020-01-01",
        transaction_costs_object=cost_dict,
        portfolio_name="GLB",
        pnl_name="PNL",
        tc_name="TCOST",
        return_pnl_excl_costs=True,
        return_costs=True,
        # concat_dfs=True,
    )

    plot_pnl(df=df_all)

```

--------------------------------

### Basic Facet Plotting Setup

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/visuals/facetplot.html

Illustrates the fundamental setup for plotting multiple tickers, optionally including a comparative series. This is used when each ticker gets its own plot.

```python
if not any([cid_grid, xcat_grid, cid_xcat_grid]):
            # each ticker gets its own plot
            for i, ticker in enumerate(tickers_to_plot):
                tks: List[str] = [ticker]
                if compare_series is not None:
                    tks.append(compare_series)
                plot_dict[i] = {
                    "X": "real_date",
                    "Y": tks,
                }
                # plot_dict[i] --> Dict[str, Union[str, List[str]]]
```

--------------------------------

### Get Start and End Dates

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/panel/make_blacklist.html

Helper function to determine the start and end dates of a sequential period within a datetime index. Used internally by `make_blacklist` to define blacklist intervals.

```python
def startend(dti, start, length):
    """
    Return start and end dates of a sequence as tuple

    Parameters
    ----------
    dti : DateTimeIndex
        datetime series of working days
    start : int
        index of start
    length : int
        number of sequential days

    Returns
    -------
    Tuple[pd.Timestamp, pd.Timestamp]
        tuple of start and end date
    """

    tup = (dti[start], dti[start + (length - 1)])
    return tup
```

--------------------------------

### Initialize DataQueryFileAPI Client

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/download/dataquery_file_api.html

Instantiate the client with optional authentication credentials and output directory. If credentials are not provided, they are sourced from environment variables.

```python
from macrosynergy.download.dataquery_file_api import DataQueryFileAPIOauth

# Example with default environment variables for credentials
client = DataQueryFileAPIOauth()

# Example with explicit credentials and output directory
client = DataQueryFileAPIOauth(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    out_dir="./custom-download-path"
)
```

--------------------------------

### Get First Usable Date for Calculations

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/pnl/historic_portfolio_volatility.html

Determines the earliest date from which calculations can reliably start, considering maximum lookback periods and nan tolerances across different frequencies. It aligns the start dates based on both pivot returns and signals.

```python
def _get_first_usable_date(
    pivot_returns: pd.DataFrame,
    pivot_signals: pd.DataFrame,
    rebal_dates: pd.Series,
    est_freqs: List[str],
    lback_periods: List[int],
    nan_tolerance: float,
) -> pd.Series:
    max_lb = 0
    # for each frequency and lookback
    for lb, est_freq in zip(lback_periods, est_freqs):
        _max_lb = get_max_lookback(lb, nan_tolerance)
        _max_lb = (
            FFILL_LIMITS[_map_to_business_day_frequency(est_freq)]
            if _max_lb == 0
            else _max_lb
        )
        max_lb = _max_lb if _max_lb > max_lb else max_lb

    assert set(pivot_returns.columns.tolist()) == set(pivot_signals.columns.tolist())
    pr_starts = {}
    ps_starts = {}
    for col in pivot_returns.columns.tolist():
        # 'full' start date for returns - where the maximum lookback period is available
        fstart_ret = pivot_returns[col].first_valid_index() + pd.offsets.BDay(max_lb)
        fstart_sig = pivot_signals[col].first_valid_index() + pd.offsets.BDay(max_lb)
        pr_starts[col] = rebal_dates[rebal_dates >= fstart_ret].min()
        ps_starts[col] = rebal_dates[rebal_dates >= fstart_sig].min()

    # get the later of the two start dates and return
    return pd.Series(
        {k: max(pr_starts[k], ps_starts[k]) for k in pr_starts.keys()},
        name="real_date",
    )
```

--------------------------------

### Initialize and Configure NaivePnL for Equity and FX

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/pnl/multi_pnl.html

Demonstrates the setup of NaivePnL objects for equity (EQXR_NSA) and foreign exchange (FXXR) data. Includes defining cross-asset categories, blacklists, and initial PnL configurations.

```python
cols_2 = cols_1 + ["ar_coef", "back_coef"]

df_xcats = pd.DataFrame(index=xcats, columns=cols_2)
df_xcats.loc["EQXR_NSA"] = ["2000-01-03", "2020-12-31", 0.1, 1, 0, 0.3]
df_xcats.loc["FXXR"] = ["2000-01-01", "2020-10-30", 1, 2, 0.95, 1]
df_xcats.loc["GROWTH"] = ["2000-01-03", "2020-10-30", 1, 2, 0.9, 1]
df_xcats.loc["INFL"] = ["2001-01-01", "2020-10-30", 1, 2, 0.8, 0.5]
df_xcats.loc["DUXR"] = ["2000-01-01", "2020-12-31", 0.1, 0.5, 0, 0.1]

black = {"AUD": ["2006-01-01", "2015-12-31"], "GBP": ["2022-01-01", "2100-01-01"]}
dfd = make_qdf(df_cids, df_xcats, back_ar=0.75)

pnl_eq = NaivePnL(
    dfd,
    ret="EQXR_NSA",
    sigs=["GROWTH"],
    cids=cids,
    start="2000-01-01",
    blacklist=black,
)

pnl_eq.make_pnl(
    sig="GROWTH",
    sig_op="zn_score_pan",
    sig_neg=False,
    sig_add=0.5,
    rebal_freq="monthly",
    vol_scale=5,
    rebal_slip=1,
    min_obs=250,
    thresh=2,
    pnl_name="PNL_EQ",
)

pnl_eq.make_long_pnl(vol_scale=10, label="LONG")

pnl_fx = NaivePnL(
    dfd,
    ret="FXXR",
    sigs=["INFL"],
    cids=cids,
    start="2000-01-01",
    blacklist=black,
)

pnl_fx.make_pnl(
    sig="INFL",
    sig_op="zn_score_pan",
    sig_neg=True,
    sig_add=0.5,
    rebal_freq="monthly",
    vol_scale=5,
    rebal_slip=1,
    min_obs=250,
    thresh=2,
    pnl_name="PNL_FX",
)

pnl_fx.make_long_pnl(vol_scale=10, label="LONG")

print(pnl_eq.pnl_names)
```

--------------------------------

### Get Common Catalog cURL Example

Source: https://docs.macrosynergy.com/stable/macrosynergy.download.fusion_interface.html

Demonstrates how to retrieve the common catalog using a cURL command. Requires an Authorization header.

```bash
curl -X GET "https://fusion.jpmorgan.com/api/v1/catalogs/common" \
    -H "Authorization: Bearer <ACCESS_TOKEN>"
```

--------------------------------

### Example Dataset Creation and Reduction

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/learning/splitters/kfold_splitters.html

Demonstrates the creation of a sample Quantamental DataFrame (QDF) and its subsequent reduction using specified parameters, including blacklisted dates.

```python
from macrosynergy.management.simulate import make_qdf
import macrosynergy.management as msm

# Example dataset
cids = ["AUD", "CAD", "GBP", "USD"]
xcats = ["XR", "CRY", "GROWTH", "INFL"]
cols = ["earliest", "latest", "mean_add", "sd_mult", "ar_coef", "back_coef"]

df_cids = pd.DataFrame(
    index=cids, columns=["earliest", "latest", "mean_add", "sd_mult"]
)
df_cids.loc["AUD"] = ["2002-01-01", "2020-12-31", 0, 1]
df_cids.loc["CAD"] = ["2003-01-01", "2020-12-31", 0, 1]
df_cids.loc["GBP"] = ["2000-01-01", "2020-12-31", 0, 1]
df_cids.loc["USD"] = ["2000-01-01", "2020-12-31", 0, 1]

df_xcats = pd.DataFrame(index=xcats, columns=cols)
df_xcats.loc["XR"] = ["2000-01-01", "2020-12-31", 0.1, 1, 0, 0.3]
df_xcats.loc["CRY"] = ["2000-01-01", "2020-12-31", 1, 2, 0.95, 1]
df_xcats.loc["GROWTH"] = ["2001-01-01", "2020-12-31", 1, 2, 0.9, 1]
df_xcats.loc["INFL"] = ["2000-01-01", "2020-12-31", 1, 2, 0.8, 0.5]

dfd = make_qdf(df_cids, df_xcats, back_ar=0.75)
dfd["grading"] = np.ones(dfd.shape[0])
black = {"GBP": ["2009-01-01", "2012-06-30"], "CAD": ["2018-01-01", "2100-01-01"]}
dfd = msm.reduce_df(df=dfd, cids=cids, xcats=xcats, blacklist=black)

dfd = dfd.pivot(index=["cid", "real_date"], columns="xcat", values="value")
X = dfd.drop(columns=["XR"])
```

--------------------------------

### Get Dataset Details cURL Example

Source: https://docs.macrosynergy.com/stable/macrosynergy.download.fusion_interface.html

Demonstrates how to retrieve details for a specific dataset within a catalog using cURL. Requires an Authorization header.

```bash
curl -X GET "https://fusion.jpmorgan.com/api/v1/catalogs/{catalog}/datasets/{dataset}" \
    -H "Authorization: Bearer <ACCESS_TOKEN>"
```

--------------------------------

### Data Query Interface Initialization and Data Download

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/download/dataquery.html

Demonstrates how to initialize the DataQueryInterface with client credentials and download data for specified expressions within a date range. It also verifies the connection before proceeding.

```python
if __name__ == "__main__":
    import os

    client_id: str = os.getenv("DQ_CLIENT_ID")
    client_secret: str = os.getenv("DQ_CLIENT_SECRET")

    expressions = ["DB(CFX,GBP,)"]

    with DataQueryInterface(
        client_id=client_id,
        client_secret=client_secret,
    ) as dq:
        assert dq.check_connection(verbose=True)

        data = dq.download_data(
            expressions=expressions,
            start_date="2024-01-25",
            end_date="2024-02-05",
            show_progress=True,
        )

    print(data)

    print(f"Succesfully downloaded data for {len(data)} expressions.")
```

--------------------------------

### Example Usage of JPMaQS Download

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/download/jpmaqs.html

Demonstrates how to instantiate and use the JPMaQSDownload class to download financial data. Ensure that DQ_CLIENT_ID and DQ_CLIENT_SECRET environment variables are set.

```python
if __name__ == "__main__":
    cids = ["AUD", "BRL", "CAD", "CHF", "CNY", "CZK", "EUR", "GBP", "USD"]
    xcats = ["RIR_NSA", "FXXR_NSA", "FXXR_VT10", "DU05YXR_NSA", "DU05YXR_VT10"]
    start_date: str = "2024-01-07"
    end_date: str = "2024-02-09"

    with JPMaQSDownload(
        client_id=os.getenv("DQ_CLIENT_ID"),
        client_secret=os.getenv("DQ_CLIENT_SECRET")
    ) as jpmaqs:
        data = jpmaqs.download(
            xcats=xcats,
            cids=cids,
            metrics="all",
            start_date=start_date,
            end_date=end_date,
            show_progress=True,
            report_time_taken=True,
        )
        print(data.info())
        print(data)
```

--------------------------------

### Get Dataset Series cURL Example

Source: https://docs.macrosynergy.com/stable/macrosynergy.download.fusion_interface.html

Demonstrates how to retrieve series available for a specific dataset within a catalog using cURL. Requires an Authorization header.

```bash
curl -X GET "https://fusion.jpmorgan.com/api/v1/catalogs/{catalog}/datasets/{dataset}/series" \
    -H "Authorization: Bearer <ACCESS_TOKEN>"
```

--------------------------------

### Connect via Proxy Server (HTTP and HTTPS)

Source: https://docs.macrosynergy.com/stable/usage_examples.html

Configure the downloader to use a proxy server for internet connections. This example shows how to set up both HTTP and HTTPS proxies.

```python
proxies = {
    "http": "http://proxy.example.com:port",
    "https": "https://secucreproxy.example.com:port",
}
with JPMaQSDownload(
        client_id = "<dq_client_id>",
        client_secret = "<dq_client_secret>",
        proxy = proxies
) as downloader:
    data = downloader.download(tickers = tickers)

```

--------------------------------

### Get Start-of-Period Dates

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/management/utils/df_utils.html

Returns a Series of start-of-period dates for a given frequency. Dates can be provided as a Series, Index, iterable, or via start and end dates.

```python
get_sops(dates=pd.to_datetime(['2023-01-15', '2023-02-20', '2023-03-10']),
             freq='M')
# Output: 
# 0   2023-01-31
# 1   2023-02-28
# 2   2023-03-31
# dtype: datetime64[ns]

get_sops(start_date='2023-01-01',
             end_date='2023-03-31',
             freq='W')
# Output: 
# DatetimeIndex(['2023-01-01', '2023-01-08', '2023-01-15', '2023-01-22',
#                '2023-01-29', '2023-02-05', '2023-02-12', '2023-02-19',
#                '2023-02-26', '2023-03-05',
#                '2023-03-12', '2023-03-19', '2023-03-26'],
#               dtype='datetime64[ns]', freq='W-SUN')
```

--------------------------------

### TransactionCosts download and get_costs example

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/pnl/transaction_costs.html

Demonstrates downloading TransactionCosts data and retrieving specific cost data for a given fid and date, then asserting its values against a test dictionary.

```python
txn_costs_obj: TransactionCosts = TransactionCosts.download()

test_dict = {
    "GBP_FXROLLCOST_MEDIAN": 0.0022470715369672,
    "GBP_FXSIZE_MEDIAN": 50.0,
    "GBP_FXSIZE_90PCTL": 200.0,
}

found_costs = txn_costs_obj.get_costs(
    fid="GBP_FX", real_date="2011-01-01"
).to_dict()

for k, v in test_dict.items():
    assert np.isclose(found_costs[k], v)
```

--------------------------------

### Example DataFrame Initialization

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/pnl/multi_pnl.html

Initializes a pandas DataFrame with specific columns and populates it with sample data for country codes (cids) and categories (xcats). This setup is used for demonstrating PnL-related functionalities.

```python
if __name__ == "__main__":
    np.random.seed(0)

    cids = ["AUD", "CAD", "GBP", "NZD", "USD", "EUR"]
    xcats = ["EQXR_NSA", "FXXR", "GROWTH", "INFL", "DUXR"]

    cols_1 = ["earliest", "latest", "mean_add", "sd_mult"]
    df_cids = pd.DataFrame(index=cids, columns=cols_1)
    df_cids.loc["AUD", :] = ["2008-01-03", "2020-12-31", 0.5, 2]
    df_cids.loc["CAD", :] = ["2010-01-03", "2020-11-30", 0, 1]
    df_cids.loc["GBP", :] = ["2012-01-03", "2020-11-30", -0.2, 0.5]
    df_cids.loc["NZD"] = ["2002-01-03", "2020-09-30", -0.1, 2]
    df_cids.loc["USD"] = ["2015-01-03", "2020-12-31", 0.2, 2]
    df_cids.loc["EUR"] = ["2008-01-03", "2020-12-31", 0.1, 2]
```

--------------------------------

### Initialize and Estimate Beta with BetaEstimator

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/learning/sequential/beta_estimator.html

Demonstrates the initialization of BetaEstimator with sample data and the subsequent estimation of beta coefficients using a Linear Regression model. This snippet shows how to set up the estimator, define models and hyperparameters, and perform the estimation with specified cross-validation and scoring.

```python
    df_cids.loc["CAD"] = ["2015-01-01", "2020-12-31", 0, 1]
    df_cids.loc["GBP"] = ["2015-01-01", "2020-12-31", 0, 1]
    df_cids.loc["USD"] = ["2015-01-01", "2020-12-31", 0, 1]

    df_xcats = pd.DataFrame(index=xcats, columns=cols)
    df_xcats.loc["BENCH_XR"] = ["2015-01-01", "2019-12-31", 0.1, 1, 0, 0.3]
    df_xcats.loc["CONTRACT_XR"] = ["2015-01-01", "2019-12-31", 0.1, 1, 0, 0.3]

    dfd = make_qdf(df_cids, df_xcats, back_ar=0.75)

    # Initialize the BetaEstimator object
    # Use for the benchmark return: USD_BENCH_XR.
    be = BetaEstimator(
        df=dfd,
        xcats="CONTRACT_XR",
        benchmark_return="USD_BENCH_XR",
        cids=["AUD", "USD"],
    )

    models = {
        "LR": LinearRegressionSystem(min_xs_samples=21 * 1),
    }
    hparam_grid = {"LR": {"fit_intercept": [True, False], "positive": [True, False]}}

    scorer = {"scorer": neg_mean_abs_corr}

    be.estimate_beta(
        beta_xcat="BETA_NSA",
        hedged_return_xcat="HEDGED_RETURN_NSA",
        models=models,
        hyperparameters=hparam_grid,
        scorers=scorer,
        inner_splitters={"expandingkfold": ExpandingKFoldPanelSplit(n_splits=5)},
        search_type="grid",
        cv_summary="median",
        min_cids=1,
        min_periods=21 * 12,
        est_freq="Q",
        n_jobs_outer=1,
        n_jobs_inner=1,
    )

    evaluation_df = be.evaluate_hedged_returns(
        correlation_types=["pearson", "spearman", "kendall"],
        freqs=["W", "M", "Q"],
    )

    be.models_heatmap(name="BETA_NSA")
```

--------------------------------

### Simulating and Preparing Panel Data for Bootstrap

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/learning/forecasting/bootstrap/bootstrap.html

Demonstrates how to simulate an unbalanced panel dataset using `make_qdf` and prepare it for bootstrapping. Includes data manipulation steps like adding a 'grading' column, applying a blacklist, and pivoting the DataFrame.

```python
if __name__ == "__main__":
    from macrosynergy.management.simulate import make_qdf
    import macrosynergy.management as msm

    # Simulate an unbalanced panel, multiindexed by cross-section and date
    cids = ["AUD", "CAD", "GBP", "USD"]
    xcats = ["XR", "CRY", "GROWTH", "INFL"]
    cols = ["earliest", "latest", "mean_add", "sd_mult", "ar_coef", "back_coef"]

    df_cids = pd.DataFrame(
        index=cids, columns=["earliest", "latest", "mean_add", "sd_mult"]
    )
    df_cids.loc["AUD"] = ["2002-01-01", "2020-12-31", 0, 1]
    df_cids.loc["CAD"] = ["2003-01-01", "2020-12-31", 0, 1]
    df_cids.loc["GBP"] = ["2000-01-01", "2020-12-31", 0, 1]
    df_cids.loc["USD"] = ["2000-01-01", "2020-12-31", 0, 1]

    df_xcats = pd.DataFrame(index=xcats, columns=cols)
    df_xcats.loc["XR"] = ["2000-01-01", "2020-12-31", 0, 1, 0, 3]
    df_xcats.loc["CRY"] = ["2000-01-01", "2020-12-31", 0, 1, 0, 0]
    df_xcats.loc["GROWTH"] = ["2000-01-01", "2020-12-31", 0, 1, -0.9, 0]
    df_xcats.loc["INFL"] = ["2000-01-01", "2020-12-31", 0, 1, 0.8, 0]

    dfd = make_qdf(df_cids, df_xcats, back_ar=0.75)
    dfd["grading"] = np.ones(dfd.shape[0])
    black = {"GBP": ["2009-01-01", "2012-06-30"], "CAD": ["2018-01-01", "2100-01-01"]}
    dfd = msm.reduce_df(df=dfd, cids=cids, xcats=xcats, blacklist=black)

    dfd = dfd.pivot(index=["cid", "real_date"], columns="xcat", values="value")
    X = dfd.drop(columns=["XR"])
    y = dfd["XR"]
```

--------------------------------

### Calculate Z-Scores for Specific Categories and Cross-Sections

Source: https://docs.macrosynergy.com/stable/_modules/macrosynergy/panel/make_zn_scores.html

This example shows how to calculate z-scores for a subset of categories and cross-sections. It specifies `xcat` and `cids` to narrow down the calculation. The `start` and `end` parameters can also be used to define a specific date range for the analysis.

```python
df = make_qdf(start_date='2010-01-01', end_date='2020-12-31', cids=list(range(4)), Партнер=list(range(2)))
zn_scores = make_zn_scores(df, xcat=['FX', 'EQ'], cids=['0', '1'], start='2015-01-01', end='2019-12-31')
print(zn_scores.head())
```
Tokens:
Raw