### Install Libraries Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Install necessary libraries for plotting and data analysis. ```shell pip install matplotlib seaborn ``` -------------------------------- ### Install algoseek-connector Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html Install the library from the Python Package Index using pip. It is recommended to do this within a virtual environment. ```bash pip install algoseek-connector ``` -------------------------------- ### Install Algoseek Connector Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Install the algoseek-connector library using pip. It is recommended to do this within a virtual environment. ```shell pip install algoseek-connector ``` -------------------------------- ### Install Plotting Libraries Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/sql.html Install matplotlib and seaborn for data visualization. These libraries are commonly used with query results. ```bash pip install matplotlib seaborn ``` -------------------------------- ### create_function_handle Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Get a FunctionHandler instance. ```APIDOC ## create_function_handle Get a FunctionHandler instance. ### Returns - **FunctionHandle** - An instance of FunctionHandle. ``` -------------------------------- ### List Available Data Sources Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Use the list_data_sources method on the ResourceManager to get a list of all accessible data sources. ```python manager.list_data_sources() ``` -------------------------------- ### DataSet.get_function_handle Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Get a handle for fast access to supported functions. ```APIDOC ## get_function_handle ### Description Get a handle for fast access to supported functions. ### Method get_function_handle ### Returns - **FunctionHandle** - A handle object for supported functions. ``` -------------------------------- ### DataSet.get_column_handle Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Get a handler object for fast access to dataset columns. ```APIDOC ## get_column_handle ### Description Get a handler object for fast access to dataset columns. ### Method get_column_handle ### Returns - **ColumnHandle** - A handler object for dataset columns. ``` -------------------------------- ### ColumnDescription Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Stores metadata for a column within a dataset, including its name, type, and description. Provides methods to get the type name, type arguments, and an HTML representation. ```APIDOC ## ColumnDescription _class _algoseek_connector.base.ColumnDescription(_name : str_, _type : str_, _description : str | None = None_) Store column metadata from a dataset. Attributes: **name: str** The column name. **type: str** The column type. **description** str, default=”” The column description Methods **get_type_name:** | Get the type name of the column. ---|--- **get_type_args:** | Get a list of type arguments. **html:** | Get an HTML representation of the column. get_type_args() → list[str] Get the type arguments. get_type_name() → str Get the type name. html() → str Create a description of the column as an HTML row. ``` -------------------------------- ### Initialize Data Source and Load Dataset Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Initialize the ResourceManager and load the US Equities Primary Exchange Daily OHLC dataset. ```python from sqlalchemy import func from algoseek_connector import ResourceManager manager = ResourceManager() ardadb = manager.create_data_source("ArdaDB") us_equity_market_data_group = ardadb.groups.USEquityMarketData.fetch() daily_ohlc = us_equity_market_data_group.datasets.PrimaryOHLCDaily.fetch() c = daily_ohlc.get_column_handle() ``` -------------------------------- ### Fetch TradeOnlyMinuteBar Data as DataFrame Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/index.rst.txt Demonstrates how to create a data source, fetch a dataset, build a query statement with filtering and ordering, and retrieve the results as a pandas DataFrame. Ensure the algoseek_connector library is imported. ```python import algoseek_connector as ac manager = ac.ResourceManager() data_source = manager.create_data_source("ArdaDB") group = data_source.groups.USEquityMarketData.fetch() dataset = group.datasets.TradeOnlyMinuteBar.fetch() stmt = ( dataset .select() .where(dataset.TradeDate > "2015-01-01"). .order_by(dataset.Volume) .limit(1000000) ) df = dataset.fetch_dataframe(stmt) ``` -------------------------------- ### Create ArdaDB Data Source Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html Instantiate an ArdaDB data source using provided credentials. Replace dummy values with your actual connection details. ```python credentials = { "host": "0.0.0.0" "port": 8123, "username": "username", "password": "password" } data_source = manager.create_data_source("ArdaDB", **credentials) ``` -------------------------------- ### Initialize ResourceManager Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html The ResourceManager is the main entry point for fetching data. Initialize it to manage available data sources. ```python import algoseek_connector as ac manager = ac.ResourceManager() ``` -------------------------------- ### DataGroupDescription Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Represents metadata for a data group, including its name, display name, and description. Provides a method to get an HTML representation. ```APIDOC ## DataGroupDescription _class _algoseek_connector.base.DataGroupDescription(_name : str_, _description : str | None = None_, _display_name : str | None = None_) Container class for datagroup metadata. Attributes: **name: str** The data group name. **display_name** str or None, default=None Name used for pretty print. **description** str or None, default=None The data group description. Methods **html:** | Get an HTML representation of the data group. ---|--- html() → str Create an HTML description of the data group. ``` -------------------------------- ### Load a Dataset Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Import the algoseek_connector library and load a dataset for querying. Ensure you have the necessary data source and group configurations. ```python import algoseek_connector as ac manager = ac.ResourceManager() data_source = manager.create_data_source("ArdaDB") group = data_source.groups.USEquityMarketData.fetch() dataset = group.datasets.TradeOnlyMinuteBar.fetch() ``` -------------------------------- ### Get Column Handle Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Obtain a column handle using the get_column_handle method for accessing dataset columns. This can also be done using attribute or index access. ```python c = dataset.get_column_handle() # access by attribute c.ASID # access by index c["ASID"] ``` -------------------------------- ### Create ArdaDB Data Source with Credentials Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Manually create an ArdaDB data source by providing database credentials. Replace dummy values with your actual connection details. ```python # dummy values used, replace with your own credentials = { "host": "0.0.0.0" "port": 8123, "username": "username", "password": "password" } data_source = manager.create_data_source("ArdaDB", **credentials) ``` -------------------------------- ### Load US Equities Primary Exchange Daily OHLC Dataset Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/sql.html Initialize ResourceManager, create an ArdaDB data source, and fetch the PrimaryOHLCDaily dataset for US Equity Market Data. ```python from sqlalchemy import func from algoseek_connector import ResourceManager manager = ResourceManager() ardadb = manager.create_data_source("ArdaDB") us_equity_market_data_group = ardadb.groups.USEquityMarketData.fetch() daily_ohlc = us_equity_market_data_group.datasets.PrimaryOHLCDaily.fetch() c = daily_ohlc.get_column_handle() ``` -------------------------------- ### Initialize ResourceManager Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Initialize the ResourceManager, which is the main entry point for interacting with data sources. Ensure the algoseek_connector library is imported. ```python import algoseek_connector as ac manager = ac.ResourceManager() ``` -------------------------------- ### Filter Data by Date Range (SQL) Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt The SQL query for filtering data within a specified date range. It selects records where the TradeDate falls between the start and end dates, inclusive. ```sql SELECT USEquityMarketData.TradeOnlyMinuteBar.TradeDate, USEquityMarketData.TradeOnlyMinuteBar.BarDateTime, USEquityMarketData.TradeOnlyMinuteBar.Ticker, USEquityMarketData.TradeOnlyMinuteBar.ASID, USEquityMarketData.TradeOnlyMinuteBar.FirstTradePrice, USEquityMarketData.TradeOnlyMinuteBar.HighTradePrice, USEquityMarketData.TradeOnlyMinuteBar.LowTradePrice, USEquityMarketData.TradeOnlyMinuteBar.LastTradePrice, USEquityMarketData.TradeOnlyMinuteBar.VolumeWeightPrice, USEquityMarketData.TradeOnlyMinuteBar.Volume, USEquityMarketData.TradeOnlyMinuteBar.TotalTrades FROM USEquityMarketData.TradeOnlyMinuteBar ``` -------------------------------- ### execute Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Execute raw SQL queries with optional parameters and output formatting. ```APIDOC ## execute Execute raw SQL queries. ### Parameters - **_sql** (str) - Parametrized SQL query. - **_parameters** (dict | None, optional) - Query parameters. Defaults to None. - **_output** (str, optional) - Output format: 'python' or 'dataframe'. Defaults to 'python'. - **_*_* kwargs** - Optional parameters passed to the `clickhouse-connect` Client.query method. ### Returns - **dict or pandas.DataFrame** - The query results in the specified output format. ``` -------------------------------- ### Fetch TradeOnlyMinuteBar data as DataFrame Source: https://algoseek-connector.readthedocs.io/en/latest/index.html This snippet demonstrates how to connect to ArdaDB, fetch USEquityMarketData, and retrieve TradeOnlyMinuteBar data as a pandas DataFrame. It includes filtering by date and ordering by volume. ```python import algoseek_connector as ac manager = ac.ResourceManager() data_source = manager.create_data_source("ArdaDB") group = data_source.groups.USEquityMarketData.fetch() dataset = group.datasets.TradeOnlyMinuteBar.fetch() stmt = ( dataset .select() .where(dataset.TradeDate > "2015-01-01"). .order_by(dataset.Volume) .limit(1000000) ) df = dataset.fetch_dataframe(stmt) ``` -------------------------------- ### ClientProtocol Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/protocols.html Adapter interface for DB clients. Defines methods for compiling SQL, creating function handles, downloading data, executing raw SQL, fetching data in various formats, and storing query results to S3. ```APIDOC ## ClientProtocol Adapter interface for DB clients. ### Methods - **compile**(_stmt : Select_) → CompiledQuery Compile a SQLAlchemy Select statement into a CompiledQuery. - **_create_function_handle**() → FunctionHandle Create a FunctionHandle instance. - **download**(_dataset : str_, _download_path : Path_, _date : date_like | tuple[date_like, date_like]_, _symbols : str | list[str]_, _expiration_date : date_like | tuple[date_like, date_like] | None_) Download data from the dataset. - **_execute**(_sql : str_, _parameters : dict | None_, _output : str_, _** kwargs_) → dict | DataFrame Execute raw SQL queries. Parameters: sql (str): Parametrized sql query. parameters (dict or None, optional): Query parameters. output ({'python', 'dataframe'}): Wether to output data using Python native types or Pandas DataFrames. **kwargs: Optional parameters passed to clickhouse-connect Client.query method. Returns: dict or pandas.DataFrame: If size is `None`. If size is provided, a generator is yield - **_fetch**(_query : CompiledQuery_, _** kwargs_) → dict[str, tuple] Fetch a select query. - **_fetch_dataframe**(_query : CompiledQuery_, _** kwargs_) → DataFrame Fetch a select query and output results as a Pandas DataFrame. - **_fetch_iter**(_query : CompiledQuery_, _size : int_, _** kwargs_) → Generator[dict[str, tuple], None, None] Yield a select query in chunks. - **_fetch_iter_dataframe**(_query : CompiledQuery_, _size : int_, _** kwargs_) → Generator[DataFrame, None, None] Yield a select query in chunks, using pandas DataFrames. - **_get_dataset_columns**(_group : str_, _name : str_) → list[Column] Create a dataset metadata instance. - **_list_datagroups**() → list[str] List available data groups. - **_list_datasets**(_group : str_) → list[str] List available datasets. - **_store_to_s3**(_query : CompiledQuery_, _bucket : str_, _key : str_, _profile_name : str | None = None_, _aws_access_key_id : str | None = None_, _aws_secret_access_key : str | None = None_) Execute a query and store results into an S3 object. Parameters: query (CompiledQuery): The query that generates the data to store on S3. bucket (str): The bucket name used to store the query. key (str): The name of the object where the query is going to be stored. profile_name (str or None, optional): If a profile name is specified, the access key and secret key are retrieved from ~/.aws/credentials and the parameters aws_access_key_id and aws_secret_access_key are ignored. If `None`, this field is ignored. aws_access_key_id (str or None, optional): The AWS access key associated with an IAM user or role. aws_secret_access_key (str or None, optional): The secret key associated with the access key. **kwargs: Key-value arguments passed to clickhouse-connect Client.query method. ``` -------------------------------- ### List Available Data Sources Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html Use the list_data_sources() method on the ResourceManager to see all data sources you have access to. ```python manager.list_data_sources() ``` -------------------------------- ### Create S3 Data Source with Credentials Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Creates an S3 data source when DB credentials are not stored in environment variables. Replace dummy values with your actual credentials. ```python # dummy values used, replace with your own credentials = { "aws_access_key_id": "aws_access_key_id", "aws_secret_access_key": "aws_secret_access_key", } data_source = manager.create_data_source("s3", **credentials) ``` -------------------------------- ### DataSet.execute Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Execute raw SQL queries against the data source. ```APIDOC ## execute ### Description Execute raw SQL queries. ### Method execute ### Parameters #### Path Parameters - **sql** (str) - Required - Parametrized SQL statement. - **parameters** (dict or None) - Optional - Query parameters. - **output** ({'python', 'dataframe'}) - Optional - Output format for query results. Defaults to 'python'. - **size** (int or None) - Optional - If a size is specified, split the results in chunks of the specified size. - **kwargs** (dict) - Optional - Extra keyword arguments passed to the underlying client. ### Returns - **dict | DataFrame** - The query results in the specified output format. ``` -------------------------------- ### compile Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Converts a `sqlalchemy.sql.selectable.Select` into a SQL string. ```APIDOC ## compile Convert a stmt into an SQL string. ### Parameters - **_stmt** (Select_) - Description of the statement to compile. - **_*_* kwargs** - Optional parameters passed to the compilation process. ``` -------------------------------- ### Set ArdaDB Credentials with Settings Class Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/configuration.rst.txt Use the AlgoseekConnectorSettings class to programmatically set ArdaDB credentials. Ensure the settings module is loaded before accessing credential fields. ```python from algoseek_connector.settings import load_settings settings = load_settings() # setting ArdaDB credentials settings.ardadb.user = "username" settings.ardadb.password = "password" ``` -------------------------------- ### Manual ArdaDB Credentials Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html When ArdaDB credentials are not in environment variables, pass them manually during data source creation. ```python ``` -------------------------------- ### S3DownloaderClient Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/api/utils.rst.txt Implementation of the ClientProtocol for S3 data sources, enabling downloading of data from S3 buckets. ```APIDOC ## S3DownloaderClient ### Description Implementation of the ClientProtocol for S3 data sources, enabling downloading of data from S3 buckets. ### Class `algoseek_connector.s3.S3DownloaderClient` ### Note This class should not be instantiated directly. Refer to the library's connection methods for proper instantiation. ### Members (Members are documented within the source class) ``` -------------------------------- ### Generate a Basic SELECT Statement Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Construct a basic SELECT statement for a dataset. The SQL tab shows the equivalent SQL query generated by the library. ```python stmt = dataset.select() ``` ```sql SELECT USEquityMarketData.TradeOnlyMinuteBar.TradeDate, USEquityMarketData.TradeOnlyMinuteBar.BarDateTime, USEquityMarketData.TradeOnlyMinuteBar.Ticker, USEquityMarketData.TradeOnlyMinuteBar.ASID, USEquityMarketData.TradeOnlyMinuteBar.FirstTradePrice, USEquityMarketData.TradeOnlyMinuteBar.HighTradePrice, USEquityMarketData.TradeOnlyMinuteBar.LowTradePrice, USEquityMarketData.TradeOnlyMinuteBar.LastTradePrice, USEquityMarketData.TradeOnlyMinuteBar.VolumeWeightPrice, USEquityMarketData.TradeOnlyMinuteBar.Volume, USEquityMarketData.TradeOnlyMinuteBar.TotalTrades FROM USEquityMarketData.TradeOnlyMinuteBar ``` -------------------------------- ### DataSetFetcher Methods Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Methods for lightweight representation of a dataset, managing data fetching and downloading. ```APIDOC ## DataSetFetcher _class _algoseek_connector.base.DataSetFetcher(_group : DataGroup_, _name : str_)¶ Lightweight representation of a dataset. Manages creation of DataSet instances for querying data using SQL and data downloading. Methods **download:** | Download data files. ---|--- **fetch:** | Create a DataSet instance. _property _description _: DataSetDescription_¶ Get the dataset name. download(_download_path : Path_, _date : date_like | tuple[date_like, date_like]_, _symbols : str | list[str]_, _expiration_date : date_like | tuple[date_like, date_like] | None = None_)¶ Download data from the dataset. Parameters: **download_path** pathlib.Path Path to a directory to download dataset files. **date** str, datetime.date or tuple Download data in this date range. Dates can be passed as a str with yyyymmdd format or as date objects. If a tuple is passed, it is interpreted as a date range and all dates in the closed interval between the two dates are generated. If a single date is passed, download data from this specific date. **symbols** str or list[str] Download data associated with these symbols. **expiration_date** str, datetime.date, tuple or None, default=None Download data with expiration dates in this date range. Dates must be passed using the same format used for the date parameter. fetch() → DataSet¶ Create a dataset instance. DataSet allow to fetch data using SQL-like queries. See here for a detailed description on how work with datasets. _property _group _: DataGroup_¶ Get the dataset group. _property _source _: DataSource_¶ Get the data source. ``` -------------------------------- ### S3DownloaderClient.list_datagroups Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html List available data groups. ```APIDOC ## list_datagroups ### Description List available data groups. ### Method ```python list_datagroups() -> list[str] ``` ``` -------------------------------- ### DescriptionProvider Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/protocols.html Interface that provides descriptions for datagroups, datasets, and columns. ```APIDOC ## DescriptionProvider Interface that provide descriptions for datagroups, datasets and columns. ### Methods - **_get_columns_description**(_group : str_, _dataset : str_) → list[ColumnDescription] Get the description of columns in a dataset. - **_get_datagroup_description**(_group : str_) → DataGroupDescription Get the description of a datagroup. - **_get_dataset_description**(_group : str_, _dataset : str_) → DataSetDescription Get the description of a dataset. ``` -------------------------------- ### download Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Not implemented. This method is intended for downloading datasets but is currently unavailable. ```APIDOC ## download Not implemented. ### Parameters - **_dataset** (str) - The name of the dataset to download. - **_download_path** (Path_) - The local path where the dataset will be saved. - **_date** (date | str | tuple[date | str, date | str]) - The date or date range for the dataset. - **_symbols** (str | list[str]) - Specific symbols to download. - **_expiration_date** (date | str | tuple[date | str, date | str]) - The expiration date for the download. ``` -------------------------------- ### Access Dataset Columns Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/sql.html Demonstrates accessing dataset columns using both attribute-style and dictionary-style access. The ColumnHandle class provides an alternative way to access columns. ```python # access by attribute col = dataset.ASID # access by index col = dataset["ASID"] ``` ```python c = dataset.get_column_handle() # access by attribute c.ASID # access by index c["ASID"] ``` -------------------------------- ### Create ArdaDB Data Source Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html Create a DataSource instance for ArdaDB using the create_data_source() method. This object will be used to interact with ArdaDB datasets. ```python data_source = manager.create_data_source("ArdaDB") ``` -------------------------------- ### S3DownloaderClient.store_to_s3 Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Download query results to S3. ```APIDOC ## store_to_s3 ### Description Download query to S3. ### Method ```python store_to_s3(query: CompiledQuery, path: str, aws_key_id: str, aws_secret_access_key: str) ``` ### Parameters #### Path Parameters - **query** (CompiledQuery) - Required - The compiled query to execute. - **path** (str) - Required - The S3 path to store the data. - **aws_key_id** (str) - Required - The AWS access key ID. - **aws_secret_access_key** (str) - Required - The AWS secret access key. ``` -------------------------------- ### Set ArdaDB Credentials with Environment Variables Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/configuration.rst.txt Configure ArdaDB credentials by exporting environment variables. The naming convention follows ALGOSEEK__{SETTINGS_GROUP}__{SETTINGS_FIELD}. ```bash export ALGOSEEK__ARDADB__USER="username" export ALGOSEEK__ARDADB__PASSWORD="password" ``` -------------------------------- ### ClickHouseClient Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/api/utils.rst.txt Implementation of the ClientProtocol for ClickHouse data sources. This client is used for interacting with ClickHouse. ```APIDOC ## ClickHouseClient ### Description Implementation of the ClientProtocol for ClickHouse data sources. This client is used for interacting with ClickHouse. ### Class `algoseek_connector.clickhouse.ClickHouseClient` ### Note This class should not be instantiated directly. Refer to the library's connection methods for proper instantiation. ### Members (Members are documented within the source class) ``` -------------------------------- ### ResourceManager Methods Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Methods for managing data sources available to a user. ```APIDOC ## ResourceManager _class _algoseek_connector.manager.ResourceManager¶ Manage data sources available to an user. Methods **create_data_source:** | Create a new DataSource instance. ---|--- **list_data_source:** | List available data sources. create_data_source(_name : str_, _** kwargs_) → DataSource¶ Create a connection to a data source. Parameters: **name** str Name of an available data source. **kwargs** dict Key-value parameters passed to the ClientProtocol used by the data source. Returns: DataSource See also `list_data_sources()` Provides a list text ids from available data sources. list_data_sources() → list[str]¶ List available data sources. ``` -------------------------------- ### SQLAlchemy Comparison Operators Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Demonstrates the use of SQLAlchemy's overloaded comparison operators for building WHERE clauses. These operators mimic standard SQL comparison logic. ```python col1 = dataset.HighTradePrice col2 = dataset.LowTradePrice, # equality col1 == col2 # greater than col1 > col2 # greater than or equal col1 >= col2 # between value1 = 1 value2 = 2 col1.between(value1, value2) # in list_of_values = [1, 2, 3, 4] col1.in_(list_of_values) ``` -------------------------------- ### ClientProtocol Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/api/protocols.rst.txt The ClientProtocol class provides the base interface for client interactions. It defines the fundamental methods that all client implementations must adhere to. ```APIDOC ## ClientProtocol ### Description The ClientProtocol class serves as the abstract base class for all client implementations. It outlines the essential methods that concrete client classes must implement to interact with the Algoseek services. ### Methods This class is intended to be subclassed, and its methods are meant to be overridden by specific client implementations. The `autoclass` directive with `:members:` indicates that all public methods of this class are documented. ``` -------------------------------- ### fetch Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Retrieve data using a select statement and return it in a Python native format. ```APIDOC ## fetch Retrieve data using a select statement. ### Parameters - **_query** (CompiledQuery_) - The query statement to fetch. - **_*_* kwargs** - Optional parameters passed to the `clickhouse-connect` Client.query method. ### Returns - **dict[str, tuple]** - A mapping from column names to values retrieved. ``` -------------------------------- ### S3Configuration Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/settings.html Configuration model for the S3 data source. ```APIDOC ## S3Configuration ### Description Store S3 data source configuration. ### Fields - **region_name** (str) - Default region when creating new connections. Defaults to 'us-east-1'. - **aws_access_key_id** (str | None) - The AWS access key id. If provided, overwrite profile credentials. - **aws_secret_access_key** (SecretStr | None) - The AWS secret access key. If provided, overwrite profile credentials. - **profile_name** (str | None) - A profile stored in ~/.aws/credentials. - **download_limit** (PositiveInt) - S3 datasets download quota, in bytes. Set by default to 1 TiB. - **download_limit_do_not_change** (PositiveInt) - A second download limit for S3 datasets, in bytes. Set by default to 20 TiB. ``` -------------------------------- ### S3DownloaderClient.download Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Download dataset files using filters for date, symbols, and expiration date. ```APIDOC ## download ### Description Download data from the dataset. ### Method ```python download(dataset_text_id: str, download_path: Path, date: date | str | tuple[date | str, date | str], symbols: str | list[str], expiration_date: date | str | tuple[date | str, date | str] | None = None) ``` ### Parameters #### Path Parameters - **dataset_text_id** (str) - Required - The dataset text id. - **download_path** (pathlib.Path) - Required - Path to a directory to download dataset files. - **date** (str, datetime.date or tuple) - Required - Download data in this date range. Dates can be passed as a str with yyyymmdd format or as date objects. If a tuple is passed, it is interpreted as a date range and all dates in the closed interval between the two dates are generated. I a single date is passed, download data from this specific date. - **symbols** (str or list[str]) - Required - Download data associated with these symbols. - **expiration_date** (str, datetime.date or tuple) - Optional - Download data with expiration dates in this date range. Dates must be passed used the same format used for the date parameter. ``` -------------------------------- ### ArdaDBConfiguration Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/settings.html Configuration model for the ArdaDB data source. ```APIDOC ## ArdaDBConfiguration ### Description Store ArdaDB data source configuration. ### Fields - **host** (str) - The ArdaDB host Address. Defaults to '0.0.0.0'. - **port** (PositiveInt) - The ArdaDB connection port. Defaults to 8123. - **user** (SecretStr) - The ArdaDB user name. - **password** (SecretStr) - The ArdaDB password. - **extra** (dict) - Optional arguments passed to clickhouse_connect.get_client. ``` -------------------------------- ### DataSource Methods Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Methods for managing the connection to a data source. ```APIDOC ## DataSource _class _algoseek_connector.base.DataSource(_client : ClientProtocol_, _description_provider : DescriptionProvider_)¶ Manage the connection to a data source. See here for a guide on how to work with data sources. Attributes: **client** ClientProtocol Provide the connection to the actual data source. **description_provider** DescriptionProvider Provide descriptions and metadata for data groups and datasets. **groups** DataGroupMapping Maintain the collection of available DataGroups. Methods **fetch_datagroup:** | Retrieve a data group from the data source. ---|--- **list_datagroups:** | List available data groups. fetch_datagroup(_name : str_) → DataGroup¶ Retrieve a data group. list_datagroups() → list[str]¶ List available data groups. ``` -------------------------------- ### Create ArdaDB Data Source Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Create a data source connection for ArdaDB using the ResourceManager. This method allows specifying the type of data source to connect to. ```python data_source = manager.create_data_source("ArdaDB") ``` -------------------------------- ### DescriptionProvider Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/api/protocols.rst.txt The DescriptionProvider interface is responsible for providing metadata and descriptions related to Algoseek services and data. It allows clients to query for information about available services, data schemas, and other relevant details. ```APIDOC ## DescriptionProvider ### Description The DescriptionProvider interface defines methods for retrieving descriptive information about the Algoseek platform. This includes details about available data, services, and their configurations. ### Methods Similar to ClientProtocol, this class is meant to be subclassed. The `:members:` option ensures that all public methods available for description retrieval are documented. ``` -------------------------------- ### DataSet.head Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Retrieve the first n rows of a dataset as a pandas DataFrame. ```APIDOC ## head ### Description Retrieve the first n rows of a dataset. ### Method head ### Parameters #### Path Parameters - **n** (int, default=10) - Optional - The number of rows to retrieve. ### Returns - **pandas.DataFrame** - A pandas DataFrame containing the first n rows. ``` -------------------------------- ### Cross-reference Data Using Reference Datasets Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Fetch and cross-reference data using lookup and master reference datasets to find the SecId for a given ticker. ```python reference_group = ardadb.groups.USEquityReferenceData.fetch() lookup = reference_group.datasets.LookupBase.fetch() lookup_stmt = lookup.select(lookup.SecId).where(lookup.Ticker == top5[0]) lookup_query_result = lookup.fetch(lookup_stmt)["SecId"] top1_secid = lookup_query_result[0] print(f"The most traded equity in 2023 SecId is {top1_secid}") master = reference_group.datasets.SecMasterBase.fetch() top1_master_stmt = master.select().where(master.SecId == top1_secid) top1_ref_data = master.fetch(top1_master_stmt) print(top1_ref_data) ``` -------------------------------- ### DatasetAPIConfiguration Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/settings.html Configuration model for the Dataset API data source. ```APIDOC ## DatasetAPIConfiguration ### Description Store dataset API configuration. ### Fields - **url** (str) - The API base URL. Defaults to 'https://datasets-metadata.algoseek.com/api/v2'. - **email** (SecretStr | None) - The email to request an access token. - **password** (SecretStr | None) - The password to request an access token. - **headers** (dict[str, str] | None) - Headers to include in all requests. ``` -------------------------------- ### List Datasets in a Data Group Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html List all available datasets within a DataGroup using the list_datasets() method. These datasets can then be queried or downloaded. ```python group.list_datasets() ``` -------------------------------- ### AlgoseekConnectorSettings Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/settings.html The main configuration model for the Algoseek connector. It aggregates settings for ArdaDB, Dataset API, and S3. These settings can be overridden using environment variables following the pattern ALGOSEEK____. ```APIDOC ## AlgoseekConnectorSettings ### Description Stores library configuration. Setting fields may be set through environment variables by using the notation `ALGOSEEK__{SETTINGS_GROUP}__{SETTINGS_FIELD}` where SETTINGS_GROUP may be ARDADB, DATASET_API or S3. ### Fields - **ardadb** (ArdaDBConfiguration) - ArdaDB settings. - **dataset_api** (DatasetAPIConfiguration) - Dataset API settings. - **s3** (S3Configuration) - S3 data source settings. ``` -------------------------------- ### fetch_dataframe Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Execute a Select statement and return data as a Pandas DataFrame. ```APIDOC ## fetch_dataframe Execute a Select statement and output data as a Pandas DataFrame. ### Parameters - **_query** (CompiledQuery_) - The query statement to fetch. - **_*_* kwargs** - Optional parameters passed to the `clickhouse-connect` Client.query_df method. ### Returns - **DataFrame** - A Pandas DataFrame containing the query results. ``` -------------------------------- ### Fetch a Dataset from a Data Group Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Instantiate a dataset fetcher for a specific dataset within a data group. This is used for querying data from ArdaDB. ```python dataset = group.datasets.TradeAndQuote.fetch() ``` -------------------------------- ### List Datasets in a Data Group Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Obtain a list of available datasets within a fetched data group. ```python group.list_datasets() ``` -------------------------------- ### DataSet.select Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Build a SQLAlchemy Select statement using method chaining. ```APIDOC ## select ### Description Create a select statement using chained methods with SQL-like syntax. ### Method select ### Parameters #### Path Parameters - **args** (tuple of Columns) - Optional - Sequence of columns included in the select statement. If no columns are provided, use all columns in the dataset. - **exclude** (Sequence[Column] or None, default=None) - Optional - List of columns to exclude from the select statement. ### Returns - **sqlalchemy.sql.selectable.Select** - A SQLAlchemy Select statement object. ``` -------------------------------- ### Fetch Data from ArdaDB Dataset Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Retrieve the first ten rows from an ArdaDB dataset using a select statement and the fetch method. This returns data as Python native objects. ```python stmt = dataset.select().limit(10) data = dataset.fetch(stmt) ``` -------------------------------- ### DataSet.fetch Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Fetch data from the data source using a SQLAlchemy Select statement. ```APIDOC ## fetch ### Description Fetch data using a select statement. ### Method fetch ### Parameters #### Path Parameters - **stmt** (Select) - Required - A SQLAlchemy Select statement created using the select method. - **kwargs** - Optional - Optional parameters passed to the underlying ClientProtocol.fetch method. ### Returns - **dict[str, tuple]** - A dictionary with column name/column data pairs. ``` -------------------------------- ### Select a Subset of Columns Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Create a SELECT statement that includes only specified columns. The SQL tab displays the corresponding SQL query. ```python stmt = dataset.select( dataset.TradeDate, dataset.Ticker, dataset.Volume, ) ``` ```sql SELECT USEquityMarketData.TradeOnlyMinuteBar.TradeDate, USEquityMarketData.TradeOnlyMinuteBar.Ticker, USEquityMarketData.TradeOnlyMinuteBar.Volume FROM USEquityMarketData.TradeOnlyMinuteBar ``` -------------------------------- ### Download Data from S3 Dataset Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Downloads data from an S3 dataset, supporting filtering by date range and symbols. Ensure the download directory exists. Be mindful of potential S3 costs. ```python from pathlib import Path group = data_source.groups.us_equity.fetch() dataset_fetcher = group.datasets.eq_taq # create download dir if it does not exists download_path = Path(".") download_path.mkdir(exist_ok=True) # set date range and symbol filters date_range = ("20230701", "20230731") symbols = ["ABC", "CDE"] dataset_fetcher.download( download_path, date=date_range, symbols=symbols ) ``` -------------------------------- ### SQLAlchemy Logical Operators for Filtering Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/sql.html Shows how to combine column conditions using logical AND (&) and OR (|) operators for more complex filtering in queries. ```python # AND col1 & col2 # OR col1 | col2 ``` -------------------------------- ### DataSet.compile Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Compiles a SQLAlchemy Select statement into a dialect-specific SQL string. ```APIDOC ## compile ### Description Compiles the statement into a dialect-specific SQL string. ### Method compile ### Parameters #### Path Parameters - **_stmt** (Select) - Required - The SQLAlchemy Select statement to compile. ### Returns - **CompiledQuery** - The compiled SQL query string. ``` -------------------------------- ### Aggregate Rows with HAVING Clause (Python) Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Aggregate rows by TradeDate and Ticker, calculating the minimum HighTradePrice and filtering for results where the minimum price is over 1000.0. Requires importing 'func' from SQLAlchemy. ```python from sqlalchemy import func c = dataset.get_column_handle() stmt = ( dataset .select( c.TradeDate, c.Ticker, func.min(c.HighTradePrice).label("MinHighPrice") ) .group_by(c.TradeDate, c.Ticker) .having(func.min(c.HighTradePrice) > 1000.0) ) ``` -------------------------------- ### DataSetDescription Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/dataset.html Stores metadata for a dataset, including its name, group, columns, and URLs for documentation and sample data. Offers methods to retrieve the table name and an HTML representation. ```APIDOC ## DataSetDescription _class _algoseek_connector.base.DataSetDescription(_name : str_, _group : str_, _columns : list[ColumnDescription]_, _display_name : str | None = None_, _description : str | None = None_, _granularity : str | None = None_, _pdf_url : str | None = None_, _sample_data_url : str | None = None_) Store data used to create dataset instances. Attributes: **name: str** The dataset name. **group: str** The datagroup name. **description: str** The dataset description. **columns: list[ColumnDescription] or None, default=None** The dataset columns. **display_name: str or None, default=None** The display name of the dataset. **granularity: str or None, default=None** The time granularity of the dataset. **pdf_url: str or None, default=None** URL to PDF documentation. **sample_data_url: str or None, default=None** Methods **get_table_name:** | Get the table name of the dataset using the notation `group.dataset`. ---|--- **html:** | Get an HTML representation of the dataset. get_table_name() → str Get the table name in the format group.name. html() → str Create an HTML description of the dataset. ``` -------------------------------- ### Fetch Data Group by Name Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html Fetch a specific DataGroup instance by its name using the fetch_datagroup() method on a DataSource object. ```python group = data_source.fetch_datagroup("USEquityData") ``` -------------------------------- ### Fetch Large Dataset in Chunks with Pandas Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/datasets.rst.txt Retrieve a large dataset in manageable chunks as pandas DataFrames. This method is memory-efficient for large query results. ```python stmt = data.select.limit(1000000) chunk_size = 100000 for df in dataset.fetch_iter_dataframe(stmt, chunk_size): print(df.head()) # do something with each data chunk... ``` -------------------------------- ### SQLAlchemy Column Comparison Operators Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/sql.html Illustrates the use of various comparison operators (equality, greater than, greater than or equal, between, in) on SQLAlchemy columns for filtering data. ```python # examples of comparison operators col1 = dataset.HighTradePrice col2 = dataset.LowTradePrice, # equality col1 == col2 # greater than col1 > col2 # greater than or equal col1 >= col2 # between value1 = 1 value2 = 2 col1.between(value1, value2) # in list_of_values = [1, 2, 3, 4] col1.in_(list_of_values) ``` -------------------------------- ### Aggregate Rows with GROUP BY in SQL Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/sql.html Standard SQL syntax for aggregating rows using GROUP BY and the AVG aggregate function. ```sql SELECT USEquityMarketData.TradeOnlyMinuteBar.TradeDate, USEquityMarketData.TradeOnlyMinuteBar.Ticker, avg(USEquityMarketData.TradeOnlyMinuteBar.HighTradePrice) AS MeanHighPrice FROM USEquityMarketData.TradeOnlyMinuteBar GROUP BY USEquityMarketData.TradeOnlyMinuteBar.TradeDate, USEquityMarketData.TradeOnlyMinuteBar.Ticker, ``` -------------------------------- ### Fetch Dataset Group and Dataset Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/tutorials/datasets.html Retrieve a specific dataset group and then a dataset within that group. Ensure the group and dataset names are correct for your data source. ```python group = data_source.groups.USEquityData.fetch() dataset = group.datasets.TradeAndQuote.fetch() ``` -------------------------------- ### DatasetAPIProvider Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Provides access to metadata API v2. By default, a timeout of 5 s is set to all requests. ```APIDOC ## class algoseek_connector.dataset_api.DatasetAPIProvider ### Description Provide access to metadata API v2. By default, a timeout of 5 s is set to all requests. ### Parameters * **config** (DatasetAPIConfiguration | None) - Optional - Configuration for the Dataset API. ### Methods * **get**(_endpoint : str_, _** kwargs_) → Response Request data using the GET method. * **endpoint** (str) - The endpoint to request data from. * **kwargs** (dict) - Keyword arguments passed to `requests.Session.get()`. * **list_datagroups**() → list[str] List available data groups. * **list_datasets**() → list[int] List all available dataset destinations. * **get_dataset**(_destination_id : int_) → DatasetVersionApiInfo Retrieve dataset destination information. * **destination_id** (int) - The dataset destination id as registered in the dataset API. * **get_data_group**(_internal_name : str_) → DataGroupApiInfo Retrieve data group information. * **internal_name** (str) - The data group internal name as registered in the dataset API. * **get_dataset_details**(_destination_id : int_) → DatasetDetails Retrieve dataset schema and long description. * **destination_id** (int) - The dataset destination id as registered in the dataset API. * **get_dataset_name**(_destination_id : int_) → str Create a unique display name for a dataset. * **destination_id** (int) - The dataset destination id as registered in the dataset API. * **get_dataset_destination_id**(_name : str_) → int Get the dataset destination id from its name. * **name** (str) - The dataset name used when creating dataset instances. ``` -------------------------------- ### list_datagroups Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html List all available data groups in the ClickHouse database. ```APIDOC ## list_datagroups List available groups. ### Returns - **list[str]** - A list of available data group names. ``` -------------------------------- ### S3DownloaderClient.list_datasets Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html List available datasets within a specific data group. ```APIDOC ## list_datasets ### Description List available data groups. ### Method ```python list_datasets(group_text_id: str) -> list[str] ``` ### Parameters #### Path Parameters - **group_text_id** (str) - Required - The data group identifier. ``` -------------------------------- ### store_to_s3 Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html Execute a query and store the results directly into an S3 object. ```APIDOC ## store_to_s3 Execute a query and store results into an S3 object. ### Parameters - **_query** (CompiledQuery_) - The query statement to fetch. - **_bucket** (str) - The bucket name used to store the query. - **_key** (str) - The name of the object where the query is going to be stored. - **_profile_name** (str | None, optional) - AWS profile name to use for authentication. Defaults to None. - **_aws_access_key_id** (str | None, optional) - AWS access key ID. Defaults to None. - **_aws_secret_access_key** (str | None, optional) - AWS secret access key. Defaults to None. - **_*_* kwargs** - Additional keyword arguments to pass to the underlying storage mechanism. ``` -------------------------------- ### ClickHouseClient Class Source: https://algoseek-connector.readthedocs.io/en/latest/algoseek_connector/api/utils.html The ClickHouseClient class provides methods to manage dataset retrieval from ClickHouse DB. ```APIDOC ## ClickHouseClient Manages dataset retrieval from ClickHouse DB. ### Methods - **create_function_handle:** Create a FunctionHandle instance. - **execute:** Execute raw SQL queries. - **download:** Not Implemented. - **fetch:** Retrieve data in Python native format using `sqlalchemy.sql.selectable.Select`. - **fetch_iter:** Retrieve data in Python native format using `sqlalchemy.sql.selectable.Select`. Stream results. - **fetch_dataframe:** Retrieve data as a Pandas DataFrame using `sqlalchemy.sql.selectable.Select`. - **fetch_iter_dataframe:** Retrieve data as a Pandas DataFrame using `sqlalchemy.sql.selectable.Select`. Stream results. - **list_datagroups:** List available data groups. - **list_datasets:** List available datasets. - **get_dataset_columns:** Create a list of `sqlalchemy.Column` for a dataset. - **compile:** Converts a `sqlalchemy.sql.selectable.Select` into a ``. - **Store_to_s3:** Store query results into a S3 object. ``` -------------------------------- ### Aggregate Rows with HAVING Clause (SQL) Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt This SQL query groups TradeOnlyMinuteBar data by TradeDate and Ticker, calculates the minimum HighTradePrice, and filters for groups where the minimum price exceeds 1000.0. ```sql SELECT USEquityMarketData.TradeOnlyMinuteBar.TradeDate, USEquityMarketData.TradeOnlyMinuteBar.Ticker, min(USEquityMarketData.TradeOnlyMinuteBar.HighTradePrice) AS MinHighPrice FROM USEquityMarketData.TradeOnlyMinuteBar GROUP BY USEquityMarketData.TradeOnlyMinuteBar.TradeDate, USEquityMarketData.TradeOnlyMinuteBar.Ticker, ``` -------------------------------- ### SQLAlchemy Logical Operators Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/tutorials/sql.rst.txt Shows how to combine conditions using SQLAlchemy's overloaded logical operators for AND, OR, and NOT. These are essential for complex filtering. ```python # AND col1 & col2 # OR col1 | col2 # NOT ~col1 ``` -------------------------------- ### DatasetAPIProvider Source: https://algoseek-connector.readthedocs.io/en/latest/_sources/algoseek_connector/api/utils.rst.txt Provides access to dataset APIs. This class is intended for direct use by consumers of the Algoseek Connector. ```APIDOC ## DatasetAPIProvider ### Description Provides access to dataset APIs. This class is intended for direct use by consumers of the Algoseek Connector. ### Class `algoseek_connector.dataset_api.DatasetAPIProvider` ### Members (Members are documented within the source class) ```