### Install Dependencies and Start Virtualenv

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/contributing.md

Install project dependencies, including test, documentation, and development extras, and activate the virtual environment using Poetry.

```bash
$ poetry install -E test -E doc -E dev
```

--------------------------------

### Instantiate Fusion Client

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/upload.md

Instantiate the Fusion client to interact with the API. Refer to the Getting Started guide for detailed setup instructions.

```python
fusion = Fusion()
```

--------------------------------

### Create synchronous FFS instance

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

Instantiates a synchronous Fusion File System client. This is the simplest way to get started with FFS.

```python
# Fusion instance setup (synchronous)
f_inst1 = Fusion()
sync_ffs = f_inst1.get_fusion_filesystem()
```

--------------------------------

### Install PyFusion from source

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Install PyFusion after downloading the source code. Ensure you have a copy of the source before running this command.

```console
pip install .
```

--------------------------------

### Install PyFusion SDK

Source: https://github.com/jpmorganchase/fusion/blob/master/README.md

Use pip to install the PyFusion SDK. Ensure you have Python and pip installed.

```bash
pip install pyfusion
```

--------------------------------

### Install PyFusion with Optional Extras

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Install PyFusion using pip. Optional extras can be installed for specific functionalities like S3, GCS, Azure support, Polars, SSE, or embeddings.

```bash
pip install pyfusion
# Optional extras
pip install "pyfusion[aws]"      # S3 support via s3fs
pip install "pyfusion[gcs]"      # GCS support via gcsfs
pip install "pyfusion[azr]"      # Azure support via adlfs
pip install "pyfusion[polars]"   # Polars DataFrame support
pip install "pyfusion[events]"   # Server-Sent Events support
pip install "pyfusion[embeddings]"  # Vector-store / OpenSearch support
pip install "pyfusion[all]"      # All optional extras
```

--------------------------------

### Retrieve Available Catalogs

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Use this method to list all available catalogs in the system. No setup is required beyond importing the SDK.

```python
fusion.list_catalogs()
```

--------------------------------

### Create asynchronous FFS instance

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

Instantiates an asynchronous Fusion File System client. Requires explicit setup of `FusionHTTPFileSystem` with client arguments.

```python
# To create an async fusion file system, will require instantiating the FusionHTTPFileSystem class itself
from fusion.fusion_filesystem import FusionHTTPFileSystem

f_inst2 = Fusion()
as_async = True

async_ffs = FusionHTTPFileSystem(
    client_kwargs={
        "root_url": f_inst2.root_url,
        "credentials": f_inst2.credentials,
    },
    asynchronous=as_async
)
```

--------------------------------

### Fusion HTTP File System Setup and Async Operations

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

Set up the Fusion HTTP File System for asynchronous operations in a Python script. Ensure a session is opened and managed within an `async with` block to prevent resource leaks.

```python
from fusion import Fusion
from fusion.fusion_filesystem import FusionHTTPFileSystem

# Basic Fusion/ffs setup
f_inst3 = Fusion()
as_async = True
async_ffs = FusionHTTPFileSystem(
    client_kwargs={
        "root_url": f_inst3.root_url,
        "credentials": f_inst3.credentials,
    },
    asynchronous=as_async
)

test_path = "common/datasets/ISS_ESG_CNTRY_RTNG_SSF/datasetseries/20250101/distributions"

# An HTTP Client session must be opened before executing any async methods on FFS
sess = await async_ffs.set_session()

# To avoid resource leaks, session should be opened in "async with" block.
# This ensures resources are cleaned up including when errors arise. Manually opening and then closing with .close() is available at user's own peril.
async with sess:
    cat_output = await async_ffs._cat(test_path)
    exists_output = await async_ffs._exists(test_path)
    target = "catalogs/common/datasets/FXO_SP/datasetseries/20230726/distributions/csv"
    async_download = await async_read(async_ffs, f_inst3, target)
```

--------------------------------

### Configure Proxies in Credentials

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Example of how to define HTTP and HTTPS proxy values in the credentials JSON file. This is necessary if your application runs behind a proxy.

```json
"proxies" : {"http": "http://proxy.myfirm.com:8080", "https": "https://proxy.myfirm.com:8080"}
```

--------------------------------

### Clone PyFusion from GitHub

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Clone the PyFusion repository from GitHub to install from source. This is an alternative installation method.

```console
git clone git://github.com/jpmorganchase/fusion
```

--------------------------------

### Import Fusion Client

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Import the Fusion class from the pyfusion library to start using the SDK.

```python
from fusion import Fusion
```

--------------------------------

### Download with Detailed Error Reporting

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Set `return_paths=True` to get more detailed output per file, including success status, local path, and error messages. This is useful for debugging unclear errors.

```python
fusion.download(
    dataset="MY_DATASET",
    dt_str="20250430",
    dataset_format="csv",
    return_paths=True
)
```

--------------------------------

### Schema Validation Example: Column Order Mismatch

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/upload.md

This example illustrates a schema validation failure due to incorrect column order. The uploaded file's column order must match the schema's defined order.

```plaintext
["id", "name", "age", "email"]
```

```plaintext
["name", "id", "email", "age"]
```

--------------------------------

### Get Async Fusion Vector Store Client

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Instantiates an asynchronous client for interacting with Fusion's vector store. Requires specifying the knowledge base and catalog names.

```python
async_client = fusion.get_async_fusion_vector_store_client(
    knowledge_base="MY_KNOWLEDGE_BASE",
    catalog="my_catalog"
)
```

--------------------------------

### Download PyFusion tarball

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Download the PyFusion source code as a tarball from GitHub. This is another alternative installation method.

```console
curl -OJL https://github.com/jpmorganchase/fusion/tarball/master
```

--------------------------------

### Schema Validation Example: Data Type Mismatch

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/upload.md

This example demonstrates a schema validation failure due to incorrect data types. The data in uploaded columns must conform to the types defined in the schema.

```plaintext
id: integer
name: string
age: integer
email: string
```

```plaintext
id,name,age,email
1,John,twenty-five,john@example.com
2,Jane,30,jane@example.com
```

--------------------------------

### Get Vector Store Client

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Returns an OpenSearch-compatible client for semantic search over indexed datasets. Requires `pyfusion[embeddings]`.

```python
from fusion import Fusion  # requires pyfusion[embeddings]

fusion = Fusion()

# Synchronous OpenSearch client
client = fusion.get_fusion_vector_store_client(
    knowledge_base="MY_KNOWLEDGE_BASE",
    catalog="my_catalog"
)
response = client.search(
    index="my_index",
    body={"query": {"match": {"text": "foreign exchange options"}}}
)
print(response["hits"]["hits"])
```

--------------------------------

### Execute Asynchronous `_cat` Operation

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

The `_cat` method, like other `_methods`, is asynchronous. To execute it in a synchronous context like a notebook, use a function like `execute_coroutine`. This example shows retrieving file content, decoding it, and parsing it.

```python
cat_coroutine = async_ffs._cat(test_path) # Returns a coroutine but does not yet execute code
cat_async_bytes = execute_coroutine(cat_coroutine)
cat_async_str = cat_async_bytes.decode("UTF-8")
cat_async_parsed = ast.literal_eval(cat_async_str)
```

--------------------------------

### Retrieve Dataset as PyArrow Table

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Use the `to_table()` method to get a dataset as a PyArrow Table. You can specify the dataset, series member, format, and catalog. Optional arguments like `columns` and `filters` allow for data selection and filtering.

```APIDOC
## to_table()

### Description
Retrieves a dataset as a PyArrow Table.

### Syntax
```python
table = fusion.to_table(
    dataset: str, 
    dt_str: str = "latest", 
    dataset_format: str = "parquet", 
    catalog: str = "common",
    columns: list[str] = None,
    filters: list[tuple] = None
)
```

### Parameters
- **dataset** (str) - Required - The dataset identifier.
- **dt_str** (str) - Optional - The specific series member of the dataset to retrieve. Defaults to "latest".
- **dataset_format** (str) - Optional - The file format (e.g., "parquet", "csv"). Defaults to "parquet".
- **catalog** (str) - Optional - The catalog identifier. Defaults to "common".
- **columns** (list[str]) - Optional - Specify which columns to include in the returned PyArrow Table.
- **filters** (list[tuple]) - Optional - Filter the data before it is returned in the PyArrow Table.

### Request Example
```python
table = fusion.to_table(
    dataset="FXO_SP",
    dt_str="2023-10-01",
    dataset_format="parquet",
    catalog="common",
    columns=["instrument_name", "fx_rate"],
    filters=[("instrument_name", "==", "AUDUSD | Spot")]
)
```

### Notes
- `to_table()` loads the dataset into a PyArrow Table, which is more memory-efficient than Pandas DataFrames.
- Consider using `filters` to reduce data size for large datasets.
- For extremely large datasets, consider `to_bytes()` or `download()`.
```

--------------------------------

### Instantiate a Product Object

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/metadata_creation.ipynb

Demonstrates how to instantiate a Product object using the Fusion SDK, specifying various attributes for the product.

```APIDOC
## Instantiate a Product Object

### Description
Instantiate a Product object with this client for metadata creation.

### Method Signature
```python
fusion.product(
    identifier: str,
    title: str,
    description: str,
    short_abstract: str,
    is_restricted: bool,
    maintainer: list,
    region: list,
    publisher: str,
    theme: str
)
```

### Parameters
*   **identifier** (str) - Required - A unique identifier for the product.
*   **title** (str) - Required - The display title of the product.
*   **description** (str) - Required - A detailed description of the product.
*   **short_abstract** (str) - Required - A brief summary of the product.
*   **is_restricted** (bool) - Required - Indicates if the product is restricted.
*   **maintainer** (list) - Required - A list of maintainers for the product.
*   **region** (list) - Required - The region(s) where the product is available.
*   **publisher** (str) - Required - The publisher of the product.
*   **theme** (str) - Required - The theme or category of the product.

### Request Example
```python
my_product = fusion.product(
    identifier="PYFUSION_PRODUCT",
    title="PyFusion Product",
    description="A product created using the PyFusion SDK.",
    short_abstract="A product created using the PyFusion SDK.",
    is_restricted=True,
    maintainer=MAINTAINER, # Assuming MAINTAINER is defined elsewhere
    region="Global",
    publisher=PUBLISHER,   # Assuming PUBLISHER is defined elsewhere
    theme="Research"
)
```

### Response
Returns an instance of the Product object with the specified attributes.
```

--------------------------------

### Instantiate Product Object

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Instantiate a Product object with this client for metadata creation.

```APIDOC
## product

### Description
Instantiate a Product object with this client for metadata creation.

### Method
Not applicable (SDK method)

### Endpoint
Not applicable (SDK method)
```

--------------------------------

### Instantiate Fusion Client with Custom Credentials Path

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Instantiate the Fusion client by providing a specific path to your credentials file.

```python
fusion = Fusion(credentials="path/to/my/credentials.json")
```

--------------------------------

### Get Distributions as DataFrame

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Gets distributions for a specified date or date range and returns the data as a dataframe.

```APIDOC
## to_df

### Description
Gets distributions for a specified date or date range and returns the data as a dataframe.

### Method
Not applicable (SDK method)

### Endpoint
Not applicable (SDK method)
```

--------------------------------

### Initialize Fusion Client and List Functionality

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/metadata_creation.ipynb

Instantiate the Fusion client to access available methods for metadata creation and management. This provides an overview of the SDK's capabilities.

```python
fusion
```

--------------------------------

### Get Distributions as Arrow Table

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Gets distributions for a specified date or date range and returns the data as an arrow table.

```APIDOC
## to_table

### Description
Gets distributions for a specified date or date range and returns the data as an arrow table.

### Method
Not applicable (SDK method)

### Endpoint
Not applicable (SDK method)
```

--------------------------------

### Instantiate Fusion Client with FusionCredentials Object

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Instantiate the Fusion client by providing credentials directly using the FusionCredentials object. Ensure you replace placeholders with your actual credentials.

```python
from fusion import FusionCredentials

credentials = FusionCredentials(
    client_id="<CLIENT_ID>",
    client_secret="<CLIENT_SECRET>",
    resource="<RESOURCE>"
)

fusion = Fusion(credentials=credentials)
```

--------------------------------

### Get Default Catalog

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Returns the default catalog.

```APIDOC
## default_catalog

### Description
Returns the default catalog.

### Method
Not applicable (SDK method)

### Endpoint
Not applicable (SDK method)
```

--------------------------------

### Clone the Project Repository

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/contributing.md

Clone your fork of the project repository locally to begin development.

```bash
$ git clone git@github.com:your_name_here/{{ cookiecutter.project_slug }}.git
```

--------------------------------

### Create a Fusion Product

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/metadata_creation.ipynb

Define and instantiate a Product object with essential metadata. Use this to represent a collection of datasets. Ensure all required fields are populated before creation.

```python
my_product = fusion.product(
    identifier="PYFUSION_PRODUCT",
    title="PyFusion Product",
    description="A product created using the PyFusion SDK.",
    short_abstract="A product created using the PyFusion SDK.",
    is_restricted=True,
    maintainer=MAINTAINER,
    region="Global",
    publisher=PUBLISHER,
    theme="Research"
)
my_product
```

--------------------------------

### Initialize Fusion Client

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Initialize the main API client using the Fusion class. Credentials can be provided via a default file, an explicit file path, or inline as a FusionCredentials object. The constructor verifies connectivity and sets up an HTTP session with automatic token refresh.

```python
from fusion import Fusion, FusionCredentials

# Option 1: default credentials file at config/client_credentials.json
fusion = Fusion()

# Option 2: explicit credentials file path
fusion = Fusion(credentials="path/to/my/credentials.json")

# Option 3: inline credentials object (no file required)
credentials = FusionCredentials(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    resource="YOUR_RESOURCE",
    auth_url="https://authe.jpmorgan.com/as/token.oauth2",
    proxies={"http": "http://proxy.myfirm.com:8080",
             "https": "https://proxy.myfirm.com:8080"},
)
fusion = Fusion(
    credentials=credentials,
    download_folder="my_downloads",
    log_level=20,   # logging.INFO
)

# Inspect all available methods
print(fusion)
```

--------------------------------

### View Available Methods

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

After instantiating the Fusion client, you can view all available methods by simply printing the client object. This will display a table listing each method and a brief description of its functionality.

```APIDOC
## View Available Methods

Once you have instantiated the client ``fusion``, running the following cell will display its available methods:

```python
fusion
```
The output will be a table containing the available methods along with a short description:
```
Fusion object 
Available methods:
+--------------------------------------+--------------------------------------------------------------------------------------------------------------+
| attribute                            | Instantiate an Attribute object with this client for metadata creation.                                      |
| attributes                           | Instantiate an Attributes object with this client for metadata creation.                                     |
| catalog_resources                    | List the resources contained within the catalog, for example products and datasets.                          |
| create_dataset_lineage               | Upload lineage to a dataset.                                                                                 |
| dataset                              | Instantiate a Dataset object with this client for metadata creation.                                         |
| dataset_resources                    | List the resources available for a dataset, currently this will always be a datasetseries.                   |
| datasetmember_resources              | List the available resources for a datasetseries member.                                                     |
| delete_all_datasetmembers            | Delete all dataset members within a dataset.                                                                 |
| delete_datasetmembers                | Delete dataset members.                                                                                      |
| download                             | Downloads the requested distributions of a dataset to disk.                                                  |
| from_bytes                           | Uploads data from an object in memory.                                                                       |
| get_async_fusion_vector_store_client | Returns Fusion Embeddings Search client.                                                                     |
| get_events                           | Run server sent event listener and print out the new events. Keyboard terminate to stop.                     |
| get_fusion_filesystem                | Retrieve Fusion file system instance.                                                                        |
| get_fusion_vector_store_client       | Returns Fusion Embeddings Search client.                                                                     |
| input_dataflow                       | Instantiate an Input Dataflow object with this client for metadata creation.                                 |
| list_catalogs                        | Lists the catalogs available to the API account.                                                             |
| list_dataset_attributes              | Returns the list of attributes that are in the dataset.                                                      |
| list_dataset_lineage                 | List the upstream and downstream lineage of the dataset.                                                     |
| list_datasetmembers                  | List the available members in the dataset series.                                                            |
| list_datasetmembers_distributions    | List the distributions of dataset members.                                                                   |
| list_datasets                        | Get the datasets contained in a catalog.                                                                     |
| list_distributions                   | List the available distributions (downloadable instances of the dataset with a format type).                 |
| list_indexes                         | List the indexes in a knowledge base.                                                                        |
| list_product_dataset_mapping         | get the product to dataset linking contained in  a catalog. A product is a grouping of datasets.             |
| list_products                        | Get the products contained in a catalog. A product is a grouping of datasets.                                |
| list_registered_attributes           | Returns the list of attributes in a catalog.                                                                 |
| listen_to_events                     | Run server sent event listener in the background. Retrieve results by running get_events.                    |
| output_dataflow                      | Instantiate an Output Dataflow object with this client for metadata creation.                                |
```
```

--------------------------------

### Load Data as Polars DataFrame

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Loads dataset distributions directly into a Polars DataFrame. Requires the 'polars' extra to be installed with pyfusion. Specify dataframe_type as 'polars'.

```python
from fusion import Fusion

fusion = Fusion()

# Load as Polars DataFrame (requires pyfusion[polars])
df_polars = fusion.to_df(
    dataset="FXO_SP",
    dt_str="2023-10-01",
    catalog="my_catalog",
    dataframe_type="polars"
)
```

--------------------------------

### Execute coroutine in synchronous context

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

A utility function to run an asynchronous coroutine within a synchronous Python environment. It gets the current event loop and runs the coroutine until completion.

```python
# Function to run async functions in a synchronous context
def execute_coroutine(coroute):
    """Execute coroutine from an un-awaited async function.
    
    Args:
        coroute (coroutine): An async function's returned coroutine.
    
    Returns:
        Result of coroutine execution.
    """
    loop = asyncio.get_event_loop()
    result = loop.run_until_complete(coroute)
    return result
```

--------------------------------

### Download Dataset with fsync

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Use fsync to download a specified dataset from Fusion to your local filesystem. Ensure you have initialized the Fusion client and obtained the Fusion and local filesystems. Replace placeholders with your actual dataset, catalog, and format IDs.

```python
from fusion import Fusion
from fusion.fs_sync import fsync
import fsspec

client = Fusion()
fs_fusion = client.get_fusion_file_system()
fs_local = fsspec.filesystem('file')

fsync(
    fs_fusion=fs_fusion,
    fs_local=fs_local,
    datasets=["<your_dataset_id>"],
    dataset_format="<your_dataset_format>",
    catalog="<your_catalog_id>",
    direction="download"
)
```

--------------------------------

### Get Entire File Asynchronously

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

Use `_async_get_file` to retrieve the entire content of a file asynchronously. Set `chunk_size` to control memory usage for large files. The output is a byte string.

```python
file_as_bytes = await f_inst1._async_get_file(target, chunk_size=1000)
char_limit = 700 # Set a display limit just so notebook doesn't have to render entire file
print(f"First 700 chars of file: {file_as_bytes[:char_limit]}")
```

--------------------------------

### Create a Dataset Object

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/metadata_creation.ipynb

Instantiate a dataset object with specified properties. Ensure all required parameters like identifier, title, and description are provided. The `is_restricted` flag controls access.

```python
my_dataset = fusion.dataset(
    identifier="PYFUSION_DATASET",
    title="PyFusion Dataset",
    description="A dataset created using the PyFusion SDK.",
    is_restricted=True,
    maintainer=MAINTAINER,
    region="Global",
    publisher=PUBLISHER,
    product="PYFUSION_PRODUCT",
    is_raw_data=False,
)
my_dataset
```

```python
my_dataset.create()
```

--------------------------------

### Create, Update, and Delete Product Metadata

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Instantiates a `Product` object to manage product metadata in the catalog. Use `.create()`, `.update()`, `.delete()`, or `.copy()` to perform operations. Products can be loaded from the catalog or from a dictionary.

```python
from fusion import Fusion

fusion = Fusion()

# Create a new product
product = fusion.product(
    identifier="MY_FX_PRODUCT",
    title="My FX Data Product",
    category="FX",
    description="FX Options spot data product.",
    region="Global",
    publisher="J.P. Morgan",
    delivery_channel="API",
    status="Available",
)
product.create(catalog="my_catalog")

# Load an existing product, modify, and update
existing = fusion.product("MY_FX_PRODUCT").from_catalog(catalog="my_catalog")
existing.title = "Updated FX Data Product"
existing.update(catalog="my_catalog")

# Copy product to another catalog
fusion.product("MY_FX_PRODUCT").copy(
    catalog_from="my_catalog",
    catalog_to="my_other_catalog"
)

# Load product metadata from a dictionary
product_dict = {
    "identifier": "MY_FX_PRODUCT",
    "title": "My FX Data Product",
    "category": "FX",
    "region": "Global",
}
product = fusion.product("MY_FX_PRODUCT").from_object(product_dict)
product.create(catalog="my_catalog")

# Delete a product
fusion.product("MY_FX_PRODUCT").delete(catalog="my_catalog")
```

--------------------------------

### List Available Distributions for a Series Member

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Use this method to check available formats for a specific series member. This helps in correctly populating the `dataset_format` argument for the `download()` method.

```python
fusion.list_distributions(
    dataset="MY_DATASET",
    series="20250430",
    catalog="my_catalog"
)
```

--------------------------------

### Deploy Project with Bump2version

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/contributing.md

Deploy the project by updating the version using bump2version (patch, minor, or major), committing changes, and pushing tags. GitHub Actions will handle the PyPI deployment.

```bash
$ poetry run bump2version patch # possible: major / minor / patch
$ git push
$ git push --tags
```

--------------------------------

### List Distributions for a File

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/get_started.ipynb

Displays available distributions (e.g., CSV, Parquet) for a given dataset and date.

```python
fusion.list_distributions("FXO_SP", "20241024")
```

--------------------------------

### Download Dataset with Default Settings

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Download a dataset using the download() method with essential parameters: dataset, dt_str, dataset_format, and catalog. Data is saved to the 'downloads' folder by default.

```python
fusion.download(
    dataset="MY_DATASET",
    dt_str="20250430",
    dataset_format="csv",
    catalog="my_catalog"
)
```

--------------------------------

### Perform Async File Read

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

Use this to read an entire file asynchronously. Ensure you have the necessary Fusion filesystem instance and target path.

```python
task_perform_async_read = async_read(async_ffs, f_inst2, target)
result = execute_coroutine(task_perform_async_read)
result
```

--------------------------------

### Browse Products within a Catalog

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Lists products available within a specified catalog. Replace `<CATALOG_ID>` with your actual catalog identifier.

```python
fusion.list_products(catalog="<CATALOG_ID>")
```

--------------------------------

### Create a Product

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/metadata_creation.ipynb

This operation creates a new product in the Fusion catalog using the instantiated Product object.

```APIDOC
## Create a Product

### Description
Creates a new product in the Fusion catalog.

### Method
```python
my_product.create()
```

### Parameters
This method does not take any parameters.

### Request Example
```python
my_product.create()
```

### Response
This operation typically returns a success status or confirmation upon successful creation of the product. Specific response details may vary.
```

--------------------------------

### Configure Fusion Client for S3

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/upload.md

Customize the Fusion client to use S3 as the file system by providing an `fsspec` S3 filesystem object during instantiation.

```python
import fsspec

my_fs = fsspec.filestsytem('s3')

fusion = Fusion(fs=my_fs)
```

--------------------------------

### Create Product in Fusion

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/metadata_creation.ipynb

Call the create method on a Product object to register it with Fusion. This action persists the product definition in the system.

```python
my_product.create()
```

--------------------------------

### Run All Tests with Tox

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/contributing.md

Ensure your changes pass all tests, including those for different Python versions, by running tox. This is a crucial step before committing.

```bash
$ poetry run tox
```

--------------------------------

### Load Data as PyArrow Table with Filtering

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Downloads and reads dataset distributions into a PyArrow Table, enabling memory-efficient columnar processing. Supports column selection and row-level filtering. Schema and row count can be inspected.

```python
from fusion import Fusion

fusion = Fusion()

table = fusion.to_table(
    dataset="FXO_SP",
    dt_str="2023-10-01",
    dataset_format="parquet",
    catalog="my_catalog",
    columns=["instrument_name", "fx_rate"],
    filters=[("instrument_name", "==", "AUDUSD | Spot")],
)
print(table.schema)
print(table.num_rows)

# Convert to pandas when needed
df = table.to_pandas()
```

--------------------------------

### Force Download and Overwrite Existing Files

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Use `force_download=True` to overwrite any existing file if the series member has been previously downloaded. This is useful for ensuring you have the latest version or if the existing file is corrupted.

```python
fusion.download(
    dataset="MY_DATASET",
    dt_str="20250430",
    dataset_format="csv",
    force_download=True
)
```

--------------------------------

### fusion.product

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Instantiates a Product object linked to the client for managing product metadata in the catalog.

```APIDOC
## fusion.product

### Description
Instantiates a `Product` object linked to the client. Call `.create()`, `.update()`, `.delete()`, or `.copy()` to manage product metadata in the catalog.

### Method Signature
`fusion.product(identifier, title, category, description, ...)`

### Parameters
- **identifier** (string) - Required - Unique identifier for the product.
- **title** (string) - Required - The title of the product.
- **category** (string) - Required - The category the product belongs to.
- **description** (string) - Optional - A description of the product.
- **region** (string) - Optional - The region associated with the product.
- **publisher** (string) - Optional - The publisher of the product.
- **delivery_channel** (string) - Optional - The delivery channel for the product.
- **status** (string) - Optional - The status of the product.

### Methods
- **create(catalog)**: Creates the product metadata in the specified catalog.
- **update(catalog)**: Updates the existing product metadata in the specified catalog.
- **delete(catalog)**: Deletes the product metadata from the specified catalog.
- **copy(catalog_from, catalog_to)**: Copies the product metadata from one catalog to another.
- **from_catalog(catalog)**: Loads an existing product from the specified catalog.
- **from_object(object)**: Instantiates a product from a dictionary or object.

### Request Example (Create Product)
```python
from fusion import Fusion

fusion = Fusion()

product = fusion.product(
    identifier="MY_FX_PRODUCT",
    title="My FX Data Product",
    category="FX",
    description="FX Options spot data product.",
    region="Global",
    publisher="J.P. Morgan",
    delivery_channel="API",
    status="Available",
)
product.create(catalog="my_catalog")
```

### Request Example (Update Product)
```python
from fusion import Fusion

fusion = Fusion()
existing = fusion.product("MY_FX_PRODUCT").from_catalog(catalog="my_catalog")
existing.title = "Updated FX Data Product"
existing.update(catalog="my_catalog")
```

### Request Example (Copy Product)
```python
from fusion import Fusion

fusion = Fusion()
fusion.product("MY_FX_PRODUCT").copy(
    catalog_from="my_catalog",
    catalog_to="my_other_catalog"
)
```

### Request Example (Load from Object)
```python
from fusion import Fusion

fusion = Fusion()
product_dict = {
    "identifier": "MY_FX_PRODUCT",
    "title": "My FX Data Product",
    "category": "FX",
    "region": "Global",
}
product = fusion.product("MY_FX_PRODUCT").from_object(product_dict)
product.create(catalog="my_catalog")
```

### Request Example (Delete Product)
```python
from fusion import Fusion

fusion = Fusion()
fusion.product("MY_FX_PRODUCT").delete(catalog="my_catalog")
```
```

--------------------------------

### Commit and Push Changes

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/contributing.md

Stage all your changes, commit them with a descriptive message, and push the branch to your GitHub repository.

```bash
$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature
```

--------------------------------

### List and Browse Products

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Retrieve a DataFrame of products within a catalog using `fusion.list_products()`. Supports filtering by substring across identifier and description, limiting results, and displaying all metadata columns.

```python
from fusion import Fusion

fusion = Fusion()

# All products in a catalog
all_products = fusion.list_products(catalog="my_catalog")

# Filter by substring (identifier or description)
fx_products = fusion.list_products(contains="FX", catalog="my_catalog")

# Filter strictly by identifier only
id_match = fusion.list_products(contains="FXO", id_contains=True, catalog="my_catalog")

# Limit results and show all metadata columns
sample = fusion.list_products(catalog="my_catalog", max_results=5, display_all_columns=True)
print(sample)
```

--------------------------------

### Initialize and Populate Benchmark Data Table

Source: https://github.com/jpmorganchase/fusion/blob/master/py_tests/bench_template.html

This JavaScript code initializes a DataTables instance for displaying benchmark results. It parses benchmark data, populates a version selection dropdown, and sets up event listeners for version changes.

```javascript
document.addEventListener('DOMContentLoaded', function () {
    const dataElement = document.getElementById('benchmark-data');
    const all_data = JSON.parse(dataElement.textContent);
    const results_table = $('#benchmark-table').DataTable({
        "searching": true,
        "ordering": true,
        "paging": true
    });
    results_table.clear();
    const versionSelect = document.getElementById('version-select');
    versionSelect.innerHTML = Object.keys(all_data).map(key => `<option value="${key}">${key}</option>`).join('');
    versionSelect.addEventListener('change', function () {
        const selectedVersion = versionSelect.value;
        const data = all_data[selectedVersion];
        populateOverview(data);
        populateBenchmarks(data.benchmarks);
    });

    function populateOverview(data) {
        document.getElementById('curr-node').innerText = data.machine_info.node;
        document.getElementById('curr-proc').innerText = data.machine_info.processor;
        document.getElementById('curr-python_version').innerText = data.machine_info.processor;
        if (data.prev === undefined) {
            document.getElementById('prev-node').innerText = 'N/A';
            document.getElementById('prev-proc').innerText = 'N/A';
            document.getElementById('prev-python_version').innerText = 'N/A';
            return;
        } else {
            document.getElementById('prev-node').innerText = data.prev.machine_info.node;
            document.getElementById('prev-proc').innerText = data.prev.machine_info.processor;
            document.getElementById('prev-python_version').innerText = data.prev.machine_info.processor;
        }
        /* document.getElementById('machine-current').innerText = data.machine_info.machine;
        document.getElementById('machine-previous').innerText = data.prev.machine_info.machine || 'N/A';
        document.getElementById('python-version-current').innerText = data.machine_info.python_version;
        document.getElementById('python-version-previous').innerText = data.prev.machine_info.python_version || 'N/A';
        document.getElementById('system-current').innerText = data.machine_info.system;
        document.getElementById('system-previous').innerText = data.prev.machine_info.system || 'N/A';
        document.getElementById('commit-id-current').innerText = data.commit_info.id;
        document.getElementById('commit-id-previous').innerText = data.prev.commit_info.id || 'N/A';
        document.getElementById('branch-current').innerText = data.commit_info.branch;
        document.getElementById('branch-previous').innerText = data.prev.commit_info.branch || 'N/A';
        document.getElementById('commit-time-current').innerText = data.commit_info.time;
        document.getElementById('commit-time-previous').innerText = data.prev.commit_info.time || 'N/A';
        document.getElementById('run-time-current').innerText = data.datetime;
        document.getElementById('run-time-previous').innerText = data.prev.datetime || 'N/A'; */
    }

    function populateBenchmarks(benchmarks) {
        results_table.clear();
        Object.entries(benchmarks).forEach(([key, benchmark]) => {
            const prevStats = benchmark.prev_stats || {};
            const formatDiff = (curr, prev) => prev ? `(${(curr - prev).toFixed(4)})` : '';
            const rowData = [
                benchmark.group || 'Ungrouped',
                benchmark.name,
                `${(benchmark.stats.mean * 1e6).toFixed(3)} ${formatDiff(benchmark.stats.mean * 1e6, prevStats.mean * 1e6)}`,
                `${(benchmark.stats.median * 1e6).toFixed(3)} ${formatDiff(benchmark.stats.median * 1e6, prevStats.median * 1e6)}`,
                `${(benchmark.stats.min * 1e6).toFixed(3)} ${formatDiff(benchmark.stats.min * 1e6, prevStats.min * 1e6)}`,
                `${(benchmark.stats.max * 1e6).toFixed(3)} ${formatDiff(benchmark.stats.max * 1e6, prevStats.max * 1e6)}`,
                `${(benchmark.stats.stddev * 1e6).toFixed(3)} ${formatDiff(benchmark.stats.stddev * 1e6, prevStats.stddev * 1e6)}`,
                `${(benchmark.stats.iqr * 1e6).toFixed(3)} ${formatDiff(benchmark.stats.iqr * 1e6, prevStats.iqr * 1e6)}`
            ];
            results_table.row.add(rowData);
        });
        results_table.draw();
    }
});
```

--------------------------------

### Browse Datasets Associated with a Product

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/quickstart.md

Lists datasets associated with a specific product within a catalog. Replace `<CATALOG_ID>` and `<PRODUCT_ID>` with your actual identifiers.

```python
fusion.list_datasets(catalog="<CATALOG_ID>", product="<PRODUCT_ID>")
```

--------------------------------

### Synchronize Datasets with Filesystem

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Bidirectionally synchronizes datasets between a local filesystem and Fusion. Uses checksums for efficient transfers and supports parallel operations.

```python
import fsspec
from fusion import Fusion
from fusion.fs_sync import fsync

client = Fusion()
fs_fusion = client.get_fusion_filesystem()
fs_local = fsspec.filesystem("file")

# Download: Fusion → local
fsync(
    fs_fusion=fs_fusion,
    fs_local=fs_local,
    datasets=["FXO_SP", "MY_DATASET"],
    dataset_format="parquet",
    catalog="my_catalog",
    direction="download",
    show_progress=True,
)
# Creates: my_catalog/FXO_SP/20231001/FXO_SP__my_catalog__20231001.parquet

# Upload: local → Fusion
fsync(
    fs_fusion=fs_fusion,
    fs_local=fs_local,
    datasets=["MY_DATASET"],
    dataset_format="csv",
    catalog="my_catalog",
    direction="upload",
    local_path="local_data/",
    show_progress=True,
)
```

--------------------------------

### Fusion Client Initialization

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Initializes the main API client for PyFusion. Supports various credential management options and configuration.

```APIDOC
## Fusion Client Initialization

`Fusion(credentials, root_url, download_folder, log_level, fs)` — authenticates and creates the main API client. Credentials can be a file path, a `FusionCredentials` object, or passed inline. The constructor verifies connectivity and sets up an HTTP session with automatic token refresh.

### Method
Constructor

### Parameters
- **credentials** (str or FusionCredentials or None) - Path to credentials file, a FusionCredentials object, or inline credentials.
- **root_url** (str or None) - The root URL for the Fusion API.
- **download_folder** (str or None) - The default folder for downloads.
- **log_level** (int or None) - The logging level for the client.
- **fs** (object or None) - Filesystem object, if not using default.

### Request Example
```python
from fusion import Fusion, FusionCredentials

# Option 1: default credentials file at config/client_credentials.json
fusion = Fusion()

# Option 2: explicit credentials file path
fusion = Fusion(credentials="path/to/my/credentials.json")

# Option 3: inline credentials object (no file required)
credentials = FusionCredentials(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    resource="YOUR_RESOURCE",
    auth_url="https://authe.jpmorgan.com/as/token.oauth2",
    proxies={"http": "http://proxy.myfirm.com:8080",
             "https": "https://proxy.myfirm.com:8080"},
)
fusion = Fusion(
    credentials=credentials,
    download_folder="my_downloads",
    log_level=20,   # logging.INFO
)

# Inspect all available methods
print(fusion)
```

### Credentials File Format
```json
{
    "client_id": "YOUR_CLIENT_ID",
    "client_secret": "YOUR_CLIENT_SECRET",
    "resource": "YOUR_RESOURCE",
    "auth_url": "https://authe.jpmorgan.com/as/token.oauth2",
    "proxies": {}
}
```
```

--------------------------------

### Create, Update, and Delete Dataset Metadata

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Instantiates a `Dataset` object to manage dataset metadata. Call `.create()`, `.update()`, or `.delete()` to manage metadata. Setting `is_raw_data=False` enables schema validation on upload.

```python
from fusion import Fusion

fusion = Fusion()

# Create a new dataset with schema validation (is_raw_data=False)
dataset = fusion.dataset(
    identifier="MY_DATASET",
    title="My FX Dataset",
    category="FX",
    description="Daily FX option spot data.",
    frequency="Daily",
    is_raw_data=False,   # enables schema validation on upload
    product="MY_FX_PRODUCT",
    region="Global",
    publisher="J.P. Morgan",
    container_type="Snapshot-Full",
    status="Available",
)
dataset.create(catalog="my_catalog")

# Update an existing dataset
existing = fusion.dataset("MY_DATASET").from_catalog(catalog="my_catalog")
existing.description = "Updated description for the FX dataset."
existing.update(catalog="my_catalog")

# Delete a dataset
fusion.dataset("MY_DATASET").delete(catalog="my_catalog")
```

--------------------------------

### Load Latest Data as Pandas DataFrame

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Downloads the latest available data for a dataset and loads it directly into a Pandas DataFrame. Requires dataset, catalog, and optionally format.

```python
from fusion import Fusion

fusion = Fusion()

# Load latest parquet as DataFrame
df = fusion.to_df(dataset="FXO_SP", catalog="my_catalog")
```

--------------------------------

### Set Session for Async FFS Methods

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

Before deploying any ffs async methods, you must first call the async method `.set_session()`. This is typically done using `execute_coroutine` in a synchronous context.

```python
sess = execute_coroutine(async_ffs.set_session())
```

--------------------------------

### Basic File Upload

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/upload.md

Upload a file to a specified dataset, catalog, and date string. Ensure the `dt_str` is in `YYYYMMDD` format for range-based downloads.

```python
fusion.upload(
    path = "path/to/my/file.csv",
    dataset= "MY_DATASET",
    dt_str= "20250430",
    catalog="my_catalog"
)
```

--------------------------------

### Display Raw, Cast, and Parsed `.cat` Output

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/fusion_filesystem.ipynb

Demonstrates the different representations of the output from the `.cat` operation: raw bytes, decoded string, and parsed dictionary.

```python
Raw .cat output <class 'bytes'>: b'{"@id":"distributions/","title":"Distributions","identifier":"distributions","resources":[{"fileExtension":".csv","mediaType":"text/csv; header=present; charset=utf-8","identifier":"csv","@id":"csv/","description":"Snapshot data will be in a tabular, comma separated format.","title":"CSV"},{"fileExtension":".parquet","mediaType":"application/parquet; header=present","identifier":"parquet","@id":"parquet/","description":"Snapshot data will be in a parquet format.","title":"Parquet"}],"description":"A list of available distributions","@context":{"@base":"https://fusion.jpmorgan.com/api/v1/","@vocab":"https://www.w3.org/ns/dcat3.jsonld"}}'
```

```python
Cast .cat output <class 'str'>: {"@id":"distributions/","title":"Distributions","identifier":"distributions","resources":[{"fileExtension":".csv","mediaType":"text/csv; header=present; charset=utf-8","identifier":"csv","@id":"csv/","description":"Snapshot data will be in a tabular, comma separated format.","title":"CSV"},{"fileExtension":".parquet","mediaType":"application/parquet; header=present","identifier":"parquet","@id":"parquet/","description":"Snapshot data will be in a parquet format.","title":"Parquet"}],"description":"A list of available distributions","@context":{"@base":"https://fusion.jpmorgan.com/api/v1/","@vocab":"https://www.w3.org/ns/dcat3.jsonld"}}
```

```python
Parsed .cat output <class 'dict'>: {'@id': 'distributions/', 'title': 'Distributions', 'identifier': 'distributions', 'resources': [{'fileExtension': '.csv', 'mediaType': 'text/csv; header=present; charset=utf-8', 'identifier': 'csv', '@id': 'csv/', 'description': 'Snapshot data will be in a tabular, comma separated format.', 'title': 'CSV'}, {'fileExtension': '.parquet', 'mediaType': 'application/parquet; header=present', 'identifier': 'parquet', '@id': 'parquet/', 'description': 'Snapshot data will be in a parquet format.', 'title': 'Parquet'}], 'description': 'A list of available distributions', '@context': {'@base': 'https://fusion.jpmorgan.com/api/v1/', '@vocab': 'https://www.w3.org/ns/dcat3.jsonld'}}
```

--------------------------------

### Loading Data as a PyArrow Table

Source: https://context7.com/jpmorganchase/fusion/llms.txt

Downloads and reads dataset distributions into a `pyarrow.Table` for memory-efficient columnar processing using `to_table`. Supports column selection and filtering.

```APIDOC
## Loading Data as a PyArrow Table

`fusion.to_table(dataset, dt_str, dataset_format, catalog, columns, filters)` — downloads and reads dataset distributions into a `pyarrow.Table` for memory-efficient columnar processing.

```python
from fusion import Fusion

fusion = Fusion()

table = fusion.to_table(
    dataset="FXO_SP",
    dt_str="2023-10-01",
    dataset_format="parquet",
    catalog="my_catalog",
    columns=["instrument_name", "fx_rate"],
    filters=[("instrument_name", "==", "AUDUSD | Spot")],
)
print(table.schema)
print(table.num_rows)

# Convert to pandas when needed
df = table.to_pandas()
```
```

--------------------------------

### Download Dataset to a Specific Folder

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Specify an alternate download location using the download_folder argument in the download() method. The folder will be created if it doesn't exist.

```python
fusion.download(
    dataset="MY_DATASET",
    dt_str="20250430",
    dataset_format="csv",
    catalog="my_catalog",
    download_folder="path/to/my/downloads"
)
```

--------------------------------

### Retrieve Dataset as PyArrow Table with Filters and Columns

Source: https://github.com/jpmorganchase/fusion/blob/master/docs/download.md

Filter and select specific columns when retrieving a dataset as a PyArrow Table using the filters and columns arguments in to_table().

```python
table = fusion.to_table(
    dataset="FXO_SP",
    dt_str="2023-10-01",
    dataset_format="parquet",
    catalog="common",
    columns=["instrument_name", "fx_rate"],
    filters=[("instrument_name", "==", "AUDUSD | Spot")],
)
```