### Install PyAthena Source: https://github.com/pyathena-dev/pyathena/blob/master/README.md Install the PyAthena library using pip. This command installs the base package. ```bash $ pip install PyAthena ``` -------------------------------- ### Install PyAthena with AioSQLAlchemy Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/introduction.md Install PyAthena with the AioSQLAlchemy extra for asynchronous database operations. ```bash pip install PyAthena[AioSQLAlchemy] ``` -------------------------------- ### Install PyAthena Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/introduction.md Install the base PyAthena package using pip. ```bash $ pip install PyAthena ``` -------------------------------- ### Install PyAthena with Arrow Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/introduction.md Install PyAthena with the Arrow extra for efficient in-memory data processing. ```bash pip install PyAthena[Arrow] ``` -------------------------------- ### Install Packages Source: https://github.com/pyathena-dev/pyathena/blob/master/benchmarks/20180915/README.rst Installs necessary packages for the benchmarking environment on Amazon Linux 2. ```bash $ sudo yum install java-1.8.0-openjdk python3 python3-pip git $ sudo pip3 install pipenv ``` -------------------------------- ### Install PyAthena with Pandas Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/introduction.md Install PyAthena with the Pandas extra for data manipulation and analysis. ```bash pip install PyAthena[Pandas] ``` -------------------------------- ### Install PyAthena with SQLAlchemy Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/introduction.md Install PyAthena with the SQLAlchemy extra for database operations and ORM capabilities. ```bash pip install PyAthena[SQLAlchemy] ``` -------------------------------- ### Install Git and Poetry Source: https://github.com/pyathena-dev/pyathena/blob/master/benchmarks/20220201/README.rst Installs git and poetry using yum and pip3 respectively. Ensure you have sudo privileges. ```bash $ sudo yum install git $ sudo pip3 install poetry ``` -------------------------------- ### Install PyAthena with Polars Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/introduction.md Install PyAthena with the Polars extra for high-performance dataframes. ```bash pip install PyAthena[Polars] ``` -------------------------------- ### Create Connection with ArrowCursor and Custom Converter (Alternative) Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md Instantiate ArrowCursor with a custom converter specified during connection creation. This is an alternative syntax to the previous example. ```python from pyathena import connect from pyathena.arrow.cursor import ArrowCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", converter=CustomArrowTypeConverter()).cursor(ArrowCursor) ``` -------------------------------- ### Check Java Version Source: https://github.com/pyathena-dev/pyathena/blob/master/benchmarks/20180915/README.rst Verifies the installed Java version. ```bash $ java -version openjdk version "1.8.0_201" OpenJDK Runtime Environment (build 1.8.0_201-b09) OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode) ``` -------------------------------- ### Run PyAthena Tests Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/testing.md Execute PyAthena tests using the 'just' task runner. Ensure 'uv' or a compatible package manager is installed. ```bash $ pip install uv or pipx install uv or brew install uv or mise install uv $ just test pyathena $ just test sqla $ just test sqla-async ``` -------------------------------- ### JSON Format ARRAY Examples Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Shows how PyAthena processes ARRAY data when it is provided or expected in JSON format. Covers both numeric and string arrays. ```text # Input: '[1, 2, 3]' # Output: [1, 2, 3] # Input: '["apple", "banana", "cherry"]' # Output: ["apple", "banana", "cherry"] ``` -------------------------------- ### STRUCT Data Format Examples Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Illustrates the input and output formats for STRUCT data, including Athena Native, JSON, and Unnamed formats, highlighting PyAthena's conversion capabilities. ```text # Athena Native Format: # Input: "{name=John, age=30}" # Output: {"name": "John", "age": 30} # JSON Format (Recommended): # Input: '{"name": "John", "age": 30}' # Output: {"name": "John", "age": 30} # Unnamed STRUCT Format: # Input: "{Alice, 25}" # Output: {"0": "Alice", "1": 25} ``` -------------------------------- ### PyAthena Async Connection String Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md For asynchronous operations, replace the driver portion (e.g., '+rest') with the async equivalent (e.g., '+aiorest'). This requires installing the async dependencies. ```text awsathena+aiorest://:@athena.{region_name}.amazonaws.com:443/{schema_name}?s3_staging_dir={s3_staging_dir}&... ``` -------------------------------- ### Athena Native ARRAY Format Examples Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Illustrates the input and output formats for Athena's native ARRAY data type. Shows how PyAthena handles arrays of integers and strings. ```text # Input: '[1, 2, 3]' # Output: [1, 2, 3] # Input: '[apple, banana, cherry]' # Output: ["apple", "banana", "cherry"] ``` -------------------------------- ### Check Python Version Source: https://github.com/pyathena-dev/pyathena/blob/master/benchmarks/20180915/README.rst Verifies the installed Python version. ```bash $ python3 -V Python 3.7.3 ``` -------------------------------- ### JSON Format MAP Data Example Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Illustrates the input and output format for MAP data when using JSON. This format is recommended for complex nested structures. ```text # Input: '{"name": "Laptop", "category": "Electronics"}' # Output: {"name": "Laptop", "category": "Electronics"} ``` -------------------------------- ### Create CloudFormation Stack for GitHub Actions OIDC Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/testing.md Example command to create a CloudFormation stack for setting up GitHub OIDC with AWS. This involves specifying parameters like GitHub organization, repository name, bucket name, role name, and workgroup name. ```bash $ aws --region us-west-2 \ cloudformation create-stack \ --stack-name github-actions-oidc-pyathena \ --capabilities CAPABILITY_NAMED_IAM \ --template-body file://./scripts/cloudformation/github_actions_oidc.yaml \ --parameters ParameterKey=GitHubOrg,ParameterValue=pyathena-dev \ ParameterKey=RepositoryName,ParameterValue=PyAthena \ ParameterKey=BucketName,ParameterValue=laughingman7743-athena \ ParameterKey=RoleName,ParameterValue=github-actions-oidc-pyathena-test \ ParameterKey=WorkGroupName,ParameterValue=pyathena-test ``` -------------------------------- ### Migrating from Raw String MAP Handling to Automatic Conversion Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Compares the 'before' and 'after' scenarios for handling MAP data retrieved from Athena. The 'after' example shows PyAthena's automatic conversion of raw strings to Python dictionaries. ```python # Before (raw string handling): result = cursor.execute("SELECT map_column FROM table").fetchone() raw_data = result[0] # "{\"key1\": \"value1\", \"key2\": \"value2\"}" import json parsed_data = json.loads(raw_data) # After (automatic conversion): result = cursor.execute("SELECT map_column FROM table").fetchone() map_data = result[0] # {"key1": "value1", "key2": "value2"} - automatically converted value = map_data['key1'] # Direct access ``` -------------------------------- ### Athena Native MAP Format Example Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Illustrates the input and output format for Athena's native MAP data type. This format is optimized for simple key-value pairs. ```text # Input: "{name=Laptop, category=Electronics}" # Output: {"name": "Laptop", "category": "Electronics"} ``` -------------------------------- ### Run PyAthena Tests with Multiple Python Versions Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/testing.md Execute tests across multiple Python versions using the 'just tox' command. Ensure 'uv' or a compatible package manager is installed. ```bash $ pip install uv or pipx install uv or brew install uv or mise install uv $ just tox ``` -------------------------------- ### PandasCursor Performance Tuning Examples Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/pandas.md Demonstrates various performance tuning options for PandasCursor, including different engines, threading, memory management, buffering, and custom data types. These options are passed as keyword arguments to the cursor.execute() method. ```python from pyathena import connect from pyathena.pandas.cursor import PandasCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=PandasCursor).cursor() # High-performance reading with PyArrow engine cursor.execute("SELECT * FROM large_table", engine="pyarrow", chunksize=100_000, use_threads=True) # Memory-conscious reading with Python engine cursor.execute("SELECT * FROM huge_table", engine="python", chunksize=25_000, low_memory=True) # Fine-tuned C engine with custom buffer cursor.execute("SELECT * FROM data_table", engine="c", chunksize=50_000, buffer_lines=100_000) # Custom data types for better performance cursor.execute("SELECT * FROM typed_table", dtype={'col1': 'int64', 'col2': 'float32'}, parse_dates=['timestamp_col']) ``` -------------------------------- ### Set Cache Size and Expiration Time Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Recommended configuration for caching, specifying both the maximum number of queries to cache and the expiration time in seconds. This example caches the last 100 queries within one hour. ```python from pyathena import connect cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor() cursor.execute("SELECT * FROM one_row", cache_size=100, cache_expiration_time=3600) # Use the last 100 queries within 1 hour as cache. ``` -------------------------------- ### Connect and Execute with AioS3FSCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Demonstrates how to establish an asynchronous connection using `aio_connect` and execute a query with `AioS3FSCursor`. It shows fetching a single row, multiple rows, and all rows from the result set. ```python from pyathena import aio_connect from pyathena.aio.s3fs.cursor import AioS3FSCursor async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor(AioS3FSCursor) await cursor.execute("SELECT * FROM many_rows") print(await cursor.fetchone()) print(await cursor.fetchmany(10)) print(await cursor.fetchall()) ``` -------------------------------- ### Execute query and access results with AsyncDictCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/cursor.md Demonstrates executing a query and iterating through results, accessing columns by name. Note: This example uses AsyncDictCursor but imports DictCursor, which might be a typo in the source. The usage pattern is correct for dictionary cursors. ```python from pyathena.connection import Connection from pyathena.cursor import DictCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor(AsyncDictCursor) query_id, future = cursor.execute("SELECT * FROM many_rows LIMIT 10") result_set = future.result() for row in result_set: print(row["a"]) ``` -------------------------------- ### Async Connection Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/aio.md Demonstrates how to establish an asynchronous connection using `aio_connect()` and utilize it as an async context manager. ```APIDOC ## Async Connection Use the `aio_connect()` function to create an async connection. It returns an `AioConnection` that produces `AioCursor` instances by default. ```python from pyathena import aio_connect conn = await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") ``` The connection supports the async context manager protocol: ```python from pyathena import aio_connect async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor() await cursor.execute("SELECT 1") print(await cursor.fetchone()) ``` ``` -------------------------------- ### Configure Partitioning and Clustering via Connection String Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Configure partitioning and clustering for columns directly in the connection string using comma-separated column names. ```text awsathena+rest://:@athena.us-west-2.amazonaws.com:443/default?partition=column1%2Ccolumn2&cluster=column1%2Ccolumn2&... ``` -------------------------------- ### Run Markdown Linting and Formatting Source: https://github.com/pyathena-dev/pyathena/blob/master/CLAUDE.md Commands to check and automatically fix markdown files using markdownlint-cli2. Ensure `mise install` has been run first to install the necessary version. ```bash just docs lint # check just docs format # auto-fix what's possible ``` -------------------------------- ### Initialize S3FSCursor with connect Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Instantiate the S3FSCursor by specifying `cursor_class` in the `connect` method. Ensure `s3_staging_dir` and `region_name` are configured. ```python from pyathena import connect from pyathena.s3fs.cursor import S3FSCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=S3FSCursor).cursor() ``` -------------------------------- ### Cancel Running Query Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/aio.md Provides an example of how to cancel a query that is currently in progress. ```APIDOC To cancel a running query: ```python from pyathena import aio_connect async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: async with conn.cursor() as cursor: await cursor.execute("SELECT * FROM many_rows") await cursor.cancel() ``` ``` -------------------------------- ### Initialize AsyncS3FSCursor with connect Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Instantiate AsyncS3FSCursor by specifying cursor_class in the connect method. Ensure s3_staging_dir and region_name are provided. ```python from pyathena import connect from pyathena.s3fs.async_cursor import AsyncS3FSCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncS3FSCursor).cursor() ``` -------------------------------- ### Initialize DictCursor with OrderedDict Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/cursor.md Configure DictCursor to use OrderedDict for results by passing dict_type=OrderedDict during connection setup. ```python from collections import OrderedDict from pyathena import connect from pyathena.cursor import DictCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=DictCursor).cursor(dict_type=OrderedDict) ``` -------------------------------- ### Check Python Version Source: https://github.com/pyathena-dev/pyathena/blob/master/benchmarks/20220201/README.rst Verifies the installed Python version. This command is useful for confirming the correct Python environment is active. ```bash python3 -V Python 3.7.10 ``` -------------------------------- ### Build Sphinx Documentation Site Source: https://github.com/pyathena-dev/pyathena/blob/master/CLAUDE.md Builds the project's Sphinx documentation site. This command should be run after ensuring markdown linting passes. ```bash just docs build # build the Sphinx site under docs/_build/html ``` -------------------------------- ### Initialize AsyncS3FSCursor via cursor method Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Instantiate AsyncS3FSCursor by passing the cursor class directly to the cursor method of a connect call. Ensure s3_staging_dir and region_name are provided. ```python from pyathena import connect from pyathena.s3fs.async_cursor import AsyncS3FSCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor(AsyncS3FSCursor) ``` -------------------------------- ### Initialize ArrowCursor with connect Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md Instantiate ArrowCursor by specifying `cursor_class=ArrowCursor` in the `connect` method. Ensure `s3_staging_dir` and `region_name` are correctly configured. ```python from pyathena import connect from pyathena.arrow.cursor import ArrowCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=ArrowCursor).cursor() ``` -------------------------------- ### Using Connection and Execute Callbacks Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Demonstrates how to use both connection-level and execute-level callbacks. The connection-level callback is set during connection creation, while the execute-level callback is provided during the execute call. Both will be invoked. ```python from pyathena import connect def connection_callback(query_id): print(f"Connection callback: {query_id}") # Log to monitoring system def execute_callback(query_id): print(f"Execute callback: {query_id}") # Store for cancellation if needed cursor = connect( s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", on_start_query_execution=connection_callback ).cursor() # This will invoke both connection_callback and execute_callback cursor.execute( "SELECT 1", on_start_query_execution=execute_callback ) ``` -------------------------------- ### SQL for Casting MAP to JSON Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md An SQL example demonstrating how to cast a MAP column to JSON format. This is recommended for handling complex nested MAP structures. ```sql SELECT CAST(complex_map AS JSON) FROM table_name ``` -------------------------------- ### Initialize AsyncPolarsCursor with connect Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Instantiate AsyncPolarsCursor by specifying cursor_class in the connect method. Ensure s3_staging_dir and region_name are configured. ```python from pyathena import connect from pyathena.polars.async_cursor import AsyncPolarsCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncPolarsCursor).cursor() ``` -------------------------------- ### Initialize PolarsCursor with Connection cursor method Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Get a PolarsCursor by calling the cursor method on a Connection object, specifying PolarsCursor. Ensure s3_staging_dir and region_name are set. ```python from pyathena.connection import Connection from pyathena.polars.cursor import PolarsCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor(PolarsCursor) ``` -------------------------------- ### Execute Query and Get Arrow Table Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md Execute a query and retrieve results directly as an Apache Arrow Table. This is useful for immediate data manipulation with Arrow. ```python from pyathena import aio_connect from pyathena.aio.arrow.cursor import AioArrowCursor async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor(AioArrowCursor) table = (await cursor.execute("SELECT * FROM many_rows")).as_arrow() print(table) print(table.column_names) print(table.num_rows) print(table.schema) ``` -------------------------------- ### Execute Query with Qmark Paramstyle Globally Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Set the global paramstyle to 'qmark' to use question mark placeholders. Parameters are provided as a list. ```python from pyathena import connect pyathena.paramstyle = "qmark" cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor() cursor.execute(""" SELECT col_string FROM one_row_complex WHERE col_string = ? """, ["'a string'"]) print(cursor.fetchall()) ``` -------------------------------- ### Convert Query Results to Arrow Table Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Use PolarsCursor to execute a query and convert the results directly into an Apache Arrow Table. This requires the PyArrow library to be installed. ```python from pyathena import connect from pyathena.polars.cursor import PolarsCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=PolarsCursor).cursor() # Convert to Arrow Table (requires pyarrow) table = cursor.execute("SELECT * FROM many_rows").as_arrow() print(table.num_rows) print(table.num_columns) print(table.schema) ``` -------------------------------- ### Defining Nested AthenaStruct Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Example of defining deeply nested STRUCT types using `AthenaStruct` for complex schemas, ensuring clear field types for nested structures. ```python AthenaStruct( ('user_id', Integer), ('profile', AthenaStruct( ('name', String), ('preferences', AthenaStruct( ('theme', String), ('language', String) )) )) ) ``` -------------------------------- ### Initialize AsyncS3FSCursor via cursor method with Connection Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Instantiate AsyncS3FSCursor by passing the cursor class directly to the cursor method of a Connection object. Ensure s3_staging_dir and region_name are provided. ```python from pyathena.connection import Connection from pyathena.s3fs.async_cursor import AsyncS3FSCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor(AsyncS3FSCursor) ``` -------------------------------- ### Execute Query and Get Polars DataFrame Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Use AioPolarsCursor to execute a query and retrieve results directly as a Polars DataFrame. This is useful for immediate data analysis and manipulation. ```python from pyathena import aio_connect from pyathena.aio.polars.cursor import AioPolarsCursor async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor(AioPolarsCursor) await cursor.execute("SELECT * FROM many_rows") df = cursor.as_polars() print(df.describe()) print(df.head()) ``` -------------------------------- ### Execute query and get future object Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Execute a SQL query using AsyncPolarsCursor. The execute method returns a tuple containing the query ID and a future object. ```python from pyathena import connect from pyathena.polars.async_cursor import AsyncPolarsCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncPolarsCursor).cursor() query_id, future = cursor.execute("SELECT * FROM many_rows") ``` -------------------------------- ### Execute query and get future object Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Execute a SQL query using the AsyncS3FSCursor. The execute method returns a tuple containing the query ID and a future object. ```python from pyathena import connect from pyathena.s3fs.async_cursor import AsyncS3FSCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncS3FSCursor).cursor() query_id, future = cursor.execute("SELECT * FROM many_rows") ``` -------------------------------- ### Configure Partitioning for a Column Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Use `awsathena_partition=True` to specify a column as a partition key for data. ```python Column("some_column", types.String, ..., awsathena_partition=True) ``` -------------------------------- ### Set Connection-Level Callback Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Configure a default callback function for all queries executed through a PyAthena connection. This function receives the query ID upon query execution start. ```python from pyathena import connect def query_callback(query_id): print(f"Query started with ID: {query_id}") # You can use query_id for monitoring or cancellation cursor = connect( s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", on_start_query_execution=query_callback ).cursor() cursor.execute("SELECT * FROM many_rows") # Callback will be invoked ``` -------------------------------- ### Read SQL Query Results as Pandas DataFrame Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/pandas.md Use `pandas.read_sql_query` to fetch query results directly into a pandas DataFrame. Ensure you have pandas installed and a PyAthena connection object. ```python from pyathena import connect import pandas as pd conn = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") df = pd.read_sql_query("SELECT * FROM many_rows", conn) print(df.head()) ``` -------------------------------- ### Execute Query and Fetch Results with AioCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/aio.md Demonstrates executing a query and fetching results using `fetchone`, `fetchmany`, and `fetchall` with an `AioCursor`. The cursor is managed using the async context manager. ```python from pyathena import aio_connect from pyathena.aio.cursor import AioCursor async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor() await cursor.execute("SELECT * FROM many_rows") print(await cursor.fetchone()) print(await cursor.fetchmany(10)) print(await cursor.fetchall()) ``` -------------------------------- ### Execute Query with Qmark Paramstyle per Execution Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Specify the 'qmark' paramstyle directly in the execute method for a single query. Parameters are provided as a list. ```python from pyathena import connect cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor() cursor.execute(""" SELECT col_string FROM one_row_complex WHERE col_string = ? """, ["'a string'"], paramstyle="qmark") print(cursor.fetchall()) ``` -------------------------------- ### Convert Arrow Results to Polars DataFrame Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md Convert the fetched Apache Arrow Table results into a Polars DataFrame using the `as_polars()` method. This requires the Polars library to be installed. ```python from pyathena import aio_connect from pyathena.aio.arrow.cursor import AioArrowCursor async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor(AioArrowCursor) await cursor.execute("SELECT * FROM many_rows") df = cursor.as_polars() ``` -------------------------------- ### Using Hive-Style Syntax for Type Hints Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Illustrates how to use Hive-style type signatures (with angle brackets and colons) directly in `result_set_type_hints`. PyAthena automatically converts these to the Trino-style syntax. ```python # Both are equivalent: result_set_type_hints={"col": "array(struct(a integer, b varchar))"} # Trino result_set_type_hints={"col": "array>"} # Hive ``` -------------------------------- ### Retrieve query results from future object Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md Call the result() method on the future object to get an AthenaArrowResultSet. This object provides access to query execution details and the actual data. ```python from pyathena import connect from pyathena.arrow.async_cursor import AsyncArrowCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncArrowCursor).cursor() query_id, future = cursor.execute("SELECT * FROM many_rows") result_set = future.result() print(result_set.state) print(result_set.state_change_reason) print(result_set.completion_date_time) print(result_set.submission_date_time) print(result_set.data_scanned_in_bytes) print(result_set.engine_execution_time_in_millis) print(result_set.query_queue_time_in_millis) print(result_set.total_execution_time_in_millis) print(result_set.query_planning_time_in_millis) print(result_set.service_processing_time_in_millis) print(result_set.output_location) print(result_set.description) for row in result_set: print(row) ``` -------------------------------- ### Fetch Query Results with AioArrowCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md Demonstrates fetching query results using fetchone, fetchmany, and fetchall methods with AioArrowCursor. Ensure the cursor is initialized with AioArrowCursor. ```python from pyathena import aio_connect from pyathena.aio.arrow.cursor import AioArrowCursor async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor(AioArrowCursor) await cursor.execute("SELECT * FROM many_rows") print(await cursor.fetchone()) print(await cursor.fetchmany()) print(await cursor.fetchall()) ``` -------------------------------- ### Initialize S3FSCursor with Connection object Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Instantiate the S3FSCursor by specifying `cursor_class` when creating a `Connection` object. Ensure `s3_staging_dir` and `region_name` are configured. ```python from pyathena.connection import Connection from pyathena.s3fs.cursor import S3FSCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=S3FSCursor).cursor() ``` -------------------------------- ### SQL Query for Casting STRUCT to JSON Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Example SQL query demonstrating how to cast a STRUCT column to JSON format, which is recommended for complex nested structures for better performance and readability. ```sql SELECT CAST(complex_struct AS JSON) FROM table_name ``` -------------------------------- ### Initialize PolarsCursor with Connection Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Instantiate a PolarsCursor using the Connection class by specifying cursor_class. Configure s3_staging_dir and region_name. ```python from pyathena.connection import Connection from pyathena.polars.cursor import PolarsCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=PolarsCursor).cursor() ``` -------------------------------- ### Fetch query results with S3FSCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Demonstrates fetching query results using `fetchone`, `fetchmany`, and `fetchall` methods after executing a query. Requires `s3_staging_dir` and `region_name` to be set. ```python from pyathena import connect from pyathena.s3fs.cursor import S3FSCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=S3FSCursor).cursor() cursor.execute("SELECT * FROM many_rows") print(cursor.fetchone()) print(cursor.fetchmany()) print(cursor.fetchall()) ``` -------------------------------- ### Enable Unload Option via cursor_kwargs Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Enable the unload option during connection setup by providing 'unload: True' within the cursor_kwargs dictionary. This is useful for handling large datasets efficiently. ```python from pyathena import connect from pyathena.polars.cursor import PolarsCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=PolarsCursor, cursor_kwargs={ "unload": True }).cursor() ``` -------------------------------- ### Initialize AsyncS3FSCursor with Connection object Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Instantiate AsyncS3FSCursor by specifying cursor_class when creating a Connection object. Ensure s3_staging_dir and region_name are provided. ```python from pyathena.connection import Connection from pyathena.s3fs.async_cursor import AsyncS3FSCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncS3FSCursor).cursor() ``` -------------------------------- ### Execute query and get future object Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md The execute method of AsyncArrowCursor returns a tuple containing the query ID and a future object. This future object can be used to retrieve the query results. ```python from pyathena import connect from pyathena.arrow.async_cursor import AsyncArrowCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncArrowCursor).cursor() query_id, future = cursor.execute("SELECT * FROM many_rows") ``` -------------------------------- ### Create and Use AioSparkCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/spark.md Demonstrates creating an AioSparkCursor and executing a simple Spark SQL query. Cursor creation must be wrapped in asyncio.to_thread. Ensure to replace 'YOUR_SPARK_WORKGROUP' with your actual workgroup. ```python import asyncio from pyathena import aio_connect from pyathena.aio.spark.cursor import AioSparkCursor async with await aio_connect(work_group="YOUR_SPARK_WORKGROUP", cursor_class=AioSparkCursor) as conn: cursor = await asyncio.to_thread(conn.cursor) await cursor.execute("""spark.sql(\"SELECT 1\").show()""") print(await cursor.get_std_out()) ``` -------------------------------- ### Run SQLAlchemy Async Dialect Tests Source: https://github.com/pyathena-dev/pyathena/blob/master/CLAUDE.md Execute tests for the asynchronous SQLAlchemy dialect of PyAthena. These tests require that linting has passed. ```bash just test sqla-async # SQLAlchemy async dialect tests ``` -------------------------------- ### Configure Partitioning and Clustering for Specific Tables via Connection String Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Specify table and column names connected by dots in the connection string to limit column options to specific tables. ```text awsathena+rest://:@athena.us-west-2.amazonaws.com:443/default?partition=table1.column1%2Ctable1.column2&cluster=table2.column1%2Ctable2.column2&... ``` -------------------------------- ### Complex Nested ARRAY Format Example Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Demonstrates the input and output format for complex, nested ARRAY data, specifically arrays containing STRUCT elements. PyAthena converts these into a list of dictionaries. ```text # Input: '[{name=John, age=30}, {name=Jane, age=25}]' # Output: [{"name": "John", "age": 30}, {"name": "Jane", "age": 25}] ``` -------------------------------- ### Handle Result Reuse Error Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Example of an error message indicating that result reuse is not enabled for the selected Athena engine version. Use a workgroup configured with Athena engine version 3. ```text pyathena.error.DatabaseError: An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: This functionality is not enabled in the selected engine version. Please check the engine version settings or contact AWS support for further assistance. ``` -------------------------------- ### Initialize AsyncPolarsCursor with Connection Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Instantiate AsyncPolarsCursor by specifying cursor_class when creating a Connection object. Ensure s3_staging_dir and region_name are configured. ```python from pyathena.connection import Connection from pyathena.polars.async_cursor import AsyncPolarsCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncPolarsCursor).cursor() ``` -------------------------------- ### Run SQLAlchemy Dialect Tests Source: https://github.com/pyathena-dev/pyathena/blob/master/CLAUDE.md Execute tests specifically for the SQLAlchemy dialect of PyAthena. Linting must pass prior to running these tests. ```bash just test sqla # SQLAlchemy dialect tests ``` -------------------------------- ### Execute query and get future object with AsyncPandasCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/pandas.md The execute method returns a tuple containing the query ID and a future object. This future object can be used to retrieve the query results asynchronously. ```python from pyathena import connect from pyathena.pandas.async_cursor import AsyncPandasCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncPandasCursor).cursor() query_id, future = cursor.execute("SELECT * FROM many_rows") ``` -------------------------------- ### Connect with Custom Polars Type Converter (Option 2) Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Specify the custom converter instance during the connection setup using the 'converter' argument. This ensures custom type mappings are used when creating the cursor. ```python from pyathena import connect from pyathena.polars.cursor import PolarsCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", converter=CustomPolarsTypeConverter()).cursor(PolarsCursor) ``` -------------------------------- ### Execute query and get DataFrame Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/pandas.md Execute a SQL query and retrieve the results directly as a pandas DataFrame using the `as_pandas()` method. The DataFrame can then be analyzed using pandas methods like `describe()` and `head()`. ```python from pyathena import connect from pyathena.pandas.cursor import PandasCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=PandasCursor).cursor() df = cursor.execute("SELECT * FROM many_rows").as_pandas() print(df.describe()) print(df.head()) ``` -------------------------------- ### Execution Options Callback for Individual Queries Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/sqlalchemy.md Specify a callback for individual queries using `execution_options`. The `on_start_query_execution` option is used to provide a function that receives the query ID. ```python from sqlalchemy import create_engine, text def specific_callback(query_id): print(f"Specific query callback: {query_id}") conn_str = "awsathena+rest://:@athena.us-west-2.amazonaws.com:443/default?s3_staging_dir=s3://YOUR_S3_BUCKET/path/to/" engine = create_engine(conn_str) with engine.connect() as connection: result = connection.execute( text("SELECT * FROM many_rows").execution_options( on_start_query_execution=specific_callback ) ) ``` -------------------------------- ### Initialize PolarsCursor with connect Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Instantiate a PolarsCursor by specifying cursor_class in the connect method. Ensure s3_staging_dir and region_name are configured. ```python from pyathena import connect from pyathena.polars.cursor import PolarsCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=PolarsCursor).cursor() ``` -------------------------------- ### Initialize AsyncPandasCursor with connect Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/pandas.md Specify AsyncPandasCursor as the cursor_class when using the connect method. Ensure s3_staging_dir and region_name are correctly set. ```python from pyathena import connect from pyathena.pandas.async_cursor import AsyncPandasCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=AsyncPandasCursor).cursor() ``` -------------------------------- ### Define a custom S3FSTypeConverter Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/s3fs.md Create a custom type converter by inheriting from `DefaultS3FSTypeConverter` and overriding methods to customize Athena data type to Python type mappings. This example shows how to add a custom type conversion. ```python from pyathena.s3fs.converter import DefaultS3FSTypeConverter from typing import Any class CustomS3FSTypeConverter(DefaultS3FSTypeConverter): def __init__(self) -> None: super().__init__() # Override specific type mappings self._mappings["custom_type"] = self._convert_custom def _convert_custom(self, value: str) -> Any: # Your custom conversion logic return value.upper() ``` -------------------------------- ### Cancel Long-Running Query with Timeout Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/usage.md Demonstrates how to use a connection-level callback to track query start and a separate thread to monitor and cancel a query if it exceeds a specified timeout. This is useful for preventing runaway analytical queries. ```python import time from concurrent.futures import ThreadPoolExecutor, TimeoutError from pyathena import connect def cancel_long_running_query(): """Example: Cancel a complex analytical query after 10 minutes.""" def track_query_start(query_id): print(f"Long-running analysis started: {query_id}") return query_id def monitor_and_cancel(cursor, timeout_minutes): """Monitor query and cancel if it exceeds timeout.""" time.sleep(timeout_minutes * 60) # Convert to seconds try: cursor.cancel() print(f"Query cancelled after {timeout_minutes} minutes timeout") except Exception as e: print(f"Cancellation failed: {e}") cursor = connect( s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", on_start_query_execution=track_query_start ).cursor() # Complex analytical query that might run for a long time long_query = """ WITH daily_metrics AS ( SELECT date_trunc('day', timestamp_col) as day, user_id, COUNT(*) as events, AVG(duration) as avg_duration FROM large_events_table WHERE timestamp_col >= current_date - interval '1' year GROUP BY 1, 2 ), user_segments AS ( SELECT user_id, CASE WHEN AVG(events) > 100 THEN 'high_activity' WHEN AVG(events) > 10 THEN 'medium_activity' ELSE 'low_activity' END as segment FROM daily_metrics GROUP BY user_id ) SELECT segment, COUNT(DISTINCT user_id) as users, AVG(events) as avg_daily_events FROM daily_metrics dm JOIN user_segments us ON dm.user_id = us.user_id GROUP BY segment ORDER BY avg_daily_events DESC """ # Use ThreadPoolExecutor for timeout management with ThreadPoolExecutor(max_workers=1) as executor: # Start timeout monitor (cancel after 10 minutes) timeout_future = executor.submit(monitor_and_cancel, cursor, 10) try: print("Starting complex analytical query (10-minute timeout)...") cursor.execute(long_query) # Process results results = cursor.fetchall() print(f"Analysis completed successfully: {len(results)} segments found") for row in results: print(f" {row[0]}: {row[1]} users, {row[2]:.1f} avg events") except Exception as e: print(f"Query failed or was cancelled: {e}") finally: # Clean up timeout monitor try: timeout_future.result(timeout=1) except TimeoutError: pass # Monitor is still running, which is fine # Run the example cancel_long_running_query() ``` -------------------------------- ### Run PyAthena Unit Tests Source: https://github.com/pyathena-dev/pyathena/blob/master/CLAUDE.md Execute unit tests for the PyAthena library. Ensure `just lint` passes before running tests. This command runs linting first. ```bash just test pyathena # Unit tests (runs lint first) ``` -------------------------------- ### Retrieve Query Execution Information Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/aio.md Accesses various attributes of the `AioCursor` after query execution to get details like query state, timings, and data scanned. These attributes provide insights into the query's performance and status. ```python from pyathena import aio_connect async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: async with conn.cursor() as cursor: await cursor.execute("SELECT * FROM many_rows") print(cursor.state) print(cursor.state_change_reason) print(cursor.completion_date_time) print(cursor.submission_date_time) print(cursor.data_scanned_in_bytes) print(cursor.engine_execution_time_in_millis) print(cursor.query_queue_time_in_millis) print(cursor.total_execution_time_in_millis) print(cursor.query_planning_time_in_millis) print(cursor.service_processing_time_in_millis) print(cursor.output_location) ``` -------------------------------- ### Fetch query results using fetchone, fetchmany, fetchall Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/pandas.md After executing a query, you can fetch results individually using `fetchone()`, in batches using `fetchmany()`, or all at once using `fetchall()`. ```python from pyathena import connect from pyathena.pandas.cursor import PandasCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=PandasCursor).cursor() cursor.execute("SELECT * FROM many_rows") print(cursor.fetchone()) print(cursor.fetchmany()) print(cursor.fetchall()) ``` -------------------------------- ### Initialize DictCursor with connect method Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/cursor.md Use DictCursor by specifying cursor_class with the connect method. Ensure s3_staging_dir and region_name are configured. ```python from pyathena import connect from pyathena.cursor import DictCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=DictCursor).cursor() ``` -------------------------------- ### Write Pandas DataFrame to SQL Table using SQLAlchemy Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/pandas.md Use `pandas.DataFrame.to_sql` with a SQLAlchemy engine to write DataFrame contents to an Amazon Athena table. Requires SQLAlchemy to be installed. Specify schema, S3 staging directory, and table location. ```python import pandas as pd from sqlalchemy import create_engine conn_str = "awsathena+rest://:@athena.{region_name}.amazonaws.com:443/" "{schema_name}?s3_staging_dir={s3_staging_dir}&location={location}&compression=snappy" engine = create_engine(conn_str.format( region_name="us-west-2", schema_name="YOUR_SCHEMA", s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", location="s3://YOUR_S3_BUCKET/path/to/" )) df = pd.DataFrame({"a": [1, 2, 3, 4, 5]}) df.to_sql("YOUR_TABLE", engine, schema="YOUR_SCHEMA", index=False, if_exists="replace", method="multi") ``` -------------------------------- ### Initialize AsyncPolarsCursor via cursor method Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Obtain an AsyncPolarsCursor by calling the cursor method on a connection object and passing AsyncPolarsCursor as an argument. Ensure s3_staging_dir and region_name are configured. ```python from pyathena import connect from pyathena.polars.async_cursor import AsyncPolarsCursor cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor(AsyncPolarsCursor) ``` -------------------------------- ### Fetch Query Results with AioPolarsCursor Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/polars.md Demonstrates fetching query results using fetchone, fetchmany, and fetchall methods with AioPolarsCursor. These methods allow for granular control over how results are retrieved. ```python from pyathena import aio_connect from pyathena.aio.polars.cursor import AioPolarsCursor async with await aio_connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2") as conn: cursor = conn.cursor(AioPolarsCursor) await cursor.execute("SELECT * FROM many_rows") print(await cursor.fetchone()) print(await cursor.fetchmany()) print(await cursor.fetchall()) ``` -------------------------------- ### Initialize ArrowCursor with Connection object Source: https://github.com/pyathena-dev/pyathena/blob/master/docs/arrow.md Instantiate ArrowCursor by specifying `cursor_class=ArrowCursor` when creating a `Connection` object. Ensure `s3_staging_dir` and `region_name` are correctly configured. ```python from pyathena.connection import Connection from pyathena.arrow.cursor import ArrowCursor cursor = Connection(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2", cursor_class=ArrowCursor).cursor() ```