### Install Dewey Data Client

Source: https://github.com/dewey-data/deweypy/blob/main/README.md

Install the package using pip.

```bash
pip install deweypy
```

--------------------------------

### Perform Async API Requests with DeweyPy

Source: https://context7.com/dewey-data/deweypy/llms.txt

Demonstrates authenticated async GET requests and concurrent execution using a shared client.

```python
import asyncio
from deweypy.auth import set_api_key
from deweypy.download import async_api_request, make_async_client

set_api_key("your_dewey_api_key_here")

async def fetch_dataset_info():
    # Simple async GET request
    response = await async_api_request(
        "GET",
        "/v1/external/data/cdst_69nriiuzp96tjqie/metadata"
    )
    metadata = response.json()
    print(f"Total files: {metadata['total_files']}")

    # Concurrent requests with shared client
    async with make_async_client() as client:
        # Fetch metadata and first page of files concurrently
        meta_task = async_api_request(
            "GET",
            "/v1/external/data/cdst_69nriiuzp96tjqie/metadata",
            client=client
        )
        files_task = async_api_request(
            "GET",
            "/v1/external/data/cdst_69nriiuzp96tjqie/files",
            params={"page": 1},
            client=client
        )
        meta_resp, files_resp = await asyncio.gather(meta_task, files_task)

        print(f"Metadata: {meta_resp.json()}")
        print(f"Files page 1: {files_resp.json()['total_files']} total files")

asyncio.run(fetch_dataset_info())
```

--------------------------------

### Low-Level Synchronous API Request

Source: https://context7.com/dewey-data/deweypy/llms.txt

Make authenticated HTTP requests to the Dewey API using api_request. Supports GET requests for metadata and file lists with query parameters. Demonstrates using a reusable client for multiple requests.

```python
from deweypy.auth import set_api_key
from deweypy.download import api_request, make_client

set_api_key("your_dewey_api_key_here")

# Simple GET request to fetch dataset metadata
response = api_request(
    "GET",
    "/v1/external/data/cdst_69nriiuzp96tjqie/metadata"
)
metadata = response.json()
print(f"Total files: {metadata['total_files']}")
print(f"Total size: {metadata['total_size']} bytes")

# Fetch paginated file list with query parameters
response = api_request(
    "GET",
    "/v1/external/data/cdst_69nriiuzp96tjqie/files",
    params={
        "page": 1,
        "partition_key_after": "2023-01-01",
        "partition_key_before": "2023-06-30"
    }
)
files_data = response.json()
print(f"Page {files_data['page']} of {files_data['total_pages']}")
for file_info in files_data['download_links'][:3]:
    print(f"  - {file_info['file_name']}: {file_info['file_size_bytes']} bytes")

# Using a reusable client for multiple requests
with make_client() as client:
    meta_resp = api_request("GET", "/v1/external/data/cdst_69nriiuzp96tjqie/metadata", client=client)
    files_resp = api_request("GET", "/v1/external/data/cdst_69nriiuzp96tjqie/files", client=client)
```

--------------------------------

### Load Data into DuckDB using Polars

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Connects to a DuckDB database and prepares to load Parquet files. This is an example of integrating Polars with DuckDB for data analysis.

```python
import duckdb

parquet_directory = "<path-to-your-parquet-files>"

con = duckdb.connect("mydata.duckdb")
```

--------------------------------

### Create Download Directory

Source: https://github.com/dewey-data/deweypy/blob/main/README.md

Create a local directory to store downloaded files.

```bash
mkdir dewey-downloads/
```

--------------------------------

### Simplified Async Download with run_speedy_download

Source: https://context7.com/dewey-data/deweypy/llms.txt

Use the run_speedy_download function for an optimized asynchronous download experience, automatically leveraging uvloop or winloop for enhanced performance. Configure download parameters like workers and chunk size.

```python
from pathlib import Path
from deweypy.auth import set_api_key
from deweypy.download import set_download_directory
from deweypy.download.speedy import run_speedy_download

# Setup
set_api_key("your_dewey_api_key_here")
set_download_directory(Path("./dewey-downloads"), auto_create=True)

# Run optimized download (automatically uses uvloop on Linux/macOS)
run_speedy_download(
    "cdst_69nriiuzp96tjqie",
    partition_key_after="2023-01-01",
    partition_key_before="2023-12-31",
    skip_existing=True,
    num_workers="auto",         # Auto-detect optimal workers
    buffer_chunk_size="auto",   # System-determined chunk size
    folder_name="my-dataset",
    max_retries=10
)
```

--------------------------------

### Initialize Synchronous DatasetDownloader

Source: https://context7.com/dewey-data/deweypy/llms.txt

Configure authentication and directory settings before using the DatasetDownloader class for concurrent file downloads.

```python
from pathlib import Path
from deweypy.auth import set_api_key
from deweypy.download import DatasetDownloader, set_download_directory

# Configure authentication and download location
set_api_key("your_dewey_api_key_here")
set_download_directory(Path("./dewey-downloads"), auto_create=True)
```

--------------------------------

### Create and Use DatasetDownloader

Source: https://context7.com/dewey-data/deweypy/llms.txt

Instantiate DatasetDownloader with options like date filtering, skipping existing files, and setting max retries. Access dataset metadata before initiating the download.

```python
downloader = DatasetDownloader(
    "cdst_69nriiuzp96tjqie",  # Dataset or folder ID
    partition_key_after="2023-01-01",   # Filter: partitions from this date
    partition_key_before="2023-12-31",  # Filter: partitions until this date
    skip_existing=True,                  # Skip already downloaded files
    max_retries=10                       # Max retry attempts per file
)

# Access dataset metadata before downloading
print(f"Total files: {downloader.metadata['total_files']}")
print(f"Total size: {downloader.metadata['total_size']} bytes")
print(f"Partition column: {downloader.metadata['partition_column']}")

# Start download with progress bar
downloader.download()
```

--------------------------------

### Run Dewey Data Client

Source: https://github.com/dewey-data/deweypy/blob/main/README.md

Execute the client to download data using your API key and folder ID.

```bash
python -m deweypy --api-key <YOUR_API_KEY> speedy-download <FOLDER_ID>
```

--------------------------------

### Perform Async Dataset Downloads via CLI

Source: https://context7.com/dewey-data/deweypy/llms.txt

Use the speedy-download command for high-performance async downloads. Supports custom worker counts, date partitioning, and retry configurations.

```bash
# Basic download with default settings (8 workers)
python -m deweypy --api-key YOUR_API_KEY speedy-download cdst_69nriiuzp96tjqie

# Download with custom worker count and download directory
python -m deweypy \
    --api-key YOUR_API_KEY \
    --download-directory ./my-downloads \
    speedy-download cdst_69nriiuzp96tjqie \
    --num-workers 16

# Download with date partition filtering (for date-partitioned datasets)
python -m deweypy \
    --api-key YOUR_API_KEY \
    speedy-download cdst_69nriiuzp96tjqie \
    --partition-key-after 2023-01-01 \
    --partition-key-before 2023-12-31

# Download with custom folder name and retry settings
python -m deweypy \
    --api-key YOUR_API_KEY \
    speedy-download cdst_69nriiuzp96tjqie \
    --folder-name my-custom-dataset \
    --max-retries 15 \
    --skip-existing

# Using environment variables for authentication
export DEWEY_API_KEY=your_api_key_here
export DEWEY_DOWNLOAD_DIRECTORY=./dewey-downloads
python -m deweypy speedy-download cdst_69nriiuzp96tjqie
```

--------------------------------

### Define Dataset Directory

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Sets up the local directory path for the downloaded dataset and verifies its existence.

```python
# These should all be properly defined based on the results above.
dewey_downloads_directory = Path(f".{os.sep}dewey-downloads")
assert dewey_downloads_directory.exists(), "Pre-condition"
# TODO: Replace with whatever your final sub-folder name ended up being.
this_dataset_sub_folder_name = "customized-monthly-patterns-example"
this_dataset_directory = dewey_downloads_directory / this_dataset_sub_folder_name
assert this_dataset_directory.exists(), "Pre-condition"

print(f"Dewey Downloads Directory: {dewey_downloads_directory}")
print(f"This Dataset Directory: {this_dataset_directory}")
```

--------------------------------

### Benchmark Download Speed via CLI

Source: https://context7.com/dewey-data/deweypy/llms.txt

Run speed-test to measure network performance by executing sequential downloads.

```bash
# Run 10 sequential downloads to measure speed
python -m deweypy --api-key YOUR_API_KEY speed-test \
    "https://api.deweydata.io/api/v1/external/data/cdst_69nriiuzp96tjqie/files" 10
```

--------------------------------

### Convert CSV to Parquet

Source: https://context7.com/dewey-data/deweypy/llms.txt

Initializes paths for converting downloaded CSV files into partitioned Parquet format.

```python
import polars as pl
from pathlib import Path

dataset_dir = Path("./dewey-downloads/my-dataset")
parquet_dir = dataset_dir / "parquet"
```

--------------------------------

### Retrieve Dataset Files with Pagination

Source: https://context7.com/dewey-data/deweypy/llms.txt

Fetches all download links for a dataset, supporting filtering by partition keys or returning a simple list of URLs.

```python
from deweypy.auth import set_api_key
from deweypy.download.synchronous import get_dataset_files

set_api_key("your_dewey_api_key_here")

# Get all files as detailed dictionaries
all_files = get_dataset_files(
    "cdst_69nriiuzp96tjqie",
    partition_key_after="2023-01-01",
    partition_key_before="2023-03-31"
)
print(f"Found {len(all_files)} files")
for file_info in all_files[:3]:
    print(f"  File: {file_info['file_name']}")
    print(f"    Size: {file_info['file_size_bytes']} bytes")
    print(f"    Partition: {file_info['partition_key']}")
    print(f"    Link: {file_info['link'][:50]}...")

# Get just the download URLs as a list
file_urls = get_dataset_files(
    "cdst_69nriiuzp96tjqie",
    to_list=True  # Returns list of URLs only
)
print(f"Download URLs: {len(file_urls)}")
```

--------------------------------

### Import Path Utilities

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Standard imports for handling file system paths.

```python
# Some useful imports
import os
from pathlib import Path
```

--------------------------------

### CLI: speedy-download

Source: https://context7.com/dewey-data/deweypy/llms.txt

High-performance asynchronous download command for datasets using HTTP/2.

```APIDOC
## CLI Command: speedy-download

### Description
Downloads datasets using async workers with HTTP/2 support. Includes features for date partitioning, custom worker counts, and automatic retries.

### Usage
python -m deweypy --api-key [API_KEY] speedy-download [DATASET_ID] [OPTIONS]

### Parameters
- **--api-key** (string) - Required - Dewey Data API key.
- **--download-directory** (string) - Optional - Path to store downloaded files.
- **--num-workers** (integer) - Optional - Number of concurrent async workers.
- **--partition-key-after** (date) - Optional - Filter for partitions after this date.
- **--partition-key-before** (date) - Optional - Filter for partitions before this date.
- **--folder-name** (string) - Optional - Custom folder name for the dataset.
- **--max-retries** (integer) - Optional - Maximum number of retries on failure.
- **--skip-existing** (flag) - Optional - Skip files that already exist in the directory.
```

--------------------------------

### Configure Download Directory Programmatically

Source: https://context7.com/dewey-data/deweypy/llms.txt

Set the storage location for downloaded files using set_download_directory, with options for automatic directory creation.

```python
from pathlib import Path
from deweypy.download import set_download_directory

# Set download directory (must exist unless auto_create=True)
download_dir = Path("./dewey-downloads")
download_dir.mkdir(parents=True, exist_ok=True)
set_download_directory(download_dir)

# Auto-create directory if it doesn't exist
set_download_directory(
    Path("./new-downloads"),
    auto_create=True,
    download_directory_source="manually_set"
)
```

--------------------------------

### Download Dataset via Shell

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Execute the download command from a terminal. Ensure the API URL and key are correctly configured before running.

```bash
# TODO (Dewey Team): Change to proper `uv run` or `uvx run` command 
python -m deweypy download {api_url}
```

--------------------------------

### Python API: Authentication and Configuration

Source: https://context7.com/dewey-data/deweypy/llms.txt

Programmatic methods to configure API keys and download directories.

```APIDOC
## Python API: Configuration

### set_api_key
Sets the Dewey API key for authentication.
- **api_key** (string) - Required - The API key.
- **api_key_source** (string) - Optional - Source identifier for debugging.

### set_download_directory
Sets the directory where downloaded files will be stored.
- **download_directory** (Path) - Required - The target directory path.
- **auto_create** (boolean) - Optional - Whether to create the directory if it does not exist.
- **download_directory_source** (string) - Optional - Source identifier for debugging.
```

--------------------------------

### Perform Synchronous Dataset Downloads via CLI

Source: https://context7.com/dewey-data/deweypy/llms.txt

Use the download command for thread-pooled synchronous downloads, suitable for environments where async is not preferred.

```bash
# Basic synchronous download
python -m deweypy --api-key YOUR_API_KEY download cdst_69nriiuzp96tjqie

# With partition filtering and retry settings
python -m deweypy \
    --api-key YOUR_API_KEY \
    download cdst_69nriiuzp96tjqie \
    --partition-key-after 2023-06-01 \
    --partition-key-before 2023-06-30 \
    --max-retries 5 \
    --skip-existing
```

--------------------------------

### Run Deweypy CLI Command

Source: https://github.com/dewey-data/deweypy/blob/main/AGENTS.md

Use this command to run the local development version of the Deweypy CLI. Ensure your virtual environment is activated.

```bash
source .venv/bin/activate
python -m deweypy hello
```

```bash
python -m deweypy --help
```

--------------------------------

### Configure API URL for Dataset Download

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Set the target API URL for the dataset. Replace the placeholder URL with the specific endpoint provided by the Dewey web application.

```python
from __future__ import annotations

api_url = "https://api.deweydata.io/api/v1/external/data/cdst_69nriiuzp96tjqie"
# TODO: Paste your appropriate API URL, uncomment the below, and comment out or
# delete the above.
# api_url = "https://api.deweydata.io/api/v1/external/data/cdst_INSERT_YOUR_PART_HERE"
```

--------------------------------

### Compare File Sizes: Gzipped CSV vs. Partitioned Parquet

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Calculates and prints the total size of gzipped CSV files and partitioned Parquet files in gigabytes, along with the compression ratio. This helps in understanding the storage efficiency of the partitioned Parquet format.

```python
csv_size = sum(f.stat().st_size for f in Path(this_dataset_directory).glob("*.csv.gz"))
partitioned_files = list(this_dataset_parquet_directory.glob("**/*.parquet"))
parquet_size = sum(f.stat().st_size for f in partitioned_files)

print("\nFile size comparison:")
print(f"Original CSV (gzipped): {csv_size / (1024**3):.2f} GB")
print(f"Partitioned Parquet: {parquet_size / (1024**3):.2f} GB")
print(f"Compression ratio: {csv_size / parquet_size:.2f}x")
```

--------------------------------

### Configure API Authentication Programmatically

Source: https://context7.com/dewey-data/deweypy/llms.txt

Set the API key using set_api_key. The source of the key can be tracked for debugging purposes.

```python
from deweypy.auth import set_api_key

# Set API key programmatically
set_api_key("your_dewey_api_key_here")

# The API key source is tracked for debugging
# Sources: "cli_args", "cli_fallback", "environment", "manually_set"
set_api_key("your_key", api_key_source="manually_set")
```

--------------------------------

### High-Performance Async Dataset Download

Source: https://context7.com/dewey-data/deweypy/llms.txt

Utilize AsyncDatasetDownloader for high-throughput downloads with configurable async workers, buffer sizes, and custom folder names. Access async cached properties like metadata and description.

```python
import asyncio
from pathlib import Path
from deweypy.auth import set_api_key
from deweypy.download import AsyncDatasetDownloader, set_download_directory

# Configure authentication
set_api_key("your_dewey_api_key_here")
set_download_directory(Path("./dewey-downloads"), auto_create=True)

async def download_dataset():
    # Create async downloader with performance tuning options
    downloader = AsyncDatasetDownloader(
        "cdst_69nriiuzp96tjqie",
        partition_key_after="2023-01-01",
        partition_key_before="2023-12-31",
        skip_existing=True,
        num_workers=8,              # Number of async workers (default: 8)
        buffer_chunk_size=131072,   # ~128KB buffer chunks
        folder_name="my-dataset",   # Custom output folder name
        max_retries=10              # Retry attempts per file
    )

    # Access async cached properties
    metadata = await downloader.metadata
    description = await downloader.description
    print(f"Dataset: {description['dataset_name']}")
    print(f"Total files: {metadata['total_files']}")

    # Start async download
    await downloader.download_all()

# Run the async download
asyncio.run(download_dataset())
```

--------------------------------

### Configure Retry Logic for Network Requests

Source: https://context7.com/dewey-data/deweypy/llms.txt

Sets up exponential backoff for transient network failures in both synchronous and asynchronous contexts.

```python
from deweypy.download.retries import (
    RetryConfig,
    sync_retrying,
    async_retrying,
    RETRYABLE_EXCEPTIONS
)

# Create custom retry configuration
config = RetryConfig(
    max_attempts=15,           # Maximum retry attempts
    base_delay_seconds=1.0,    # Base delay for exponential backoff
    max_delay_seconds=120.0,   # Maximum delay cap
    retryable_exceptions=RETRYABLE_EXCEPTIONS  # Network errors to retry
)

# Use in synchronous code
for attempt in sync_retrying(config):
    with attempt:
        if attempt.retry_state.attempt_number > 1:
            print(f"Retry attempt {attempt.retry_state.attempt_number}")
        # Your code that might fail transiently
        response = some_network_call()

# Use in async code
async for attempt in async_retrying(config):
    with attempt:
        if attempt.retry_state.attempt_number > 1:
            print(f"Async retry attempt {attempt.retry_state.attempt_number}")
        response = await some_async_network_call()
```

--------------------------------

### Convert CSV to Parquet for Faster Querying

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

This code prepares a directory for Parquet files and is intended to be run after a Polars DataFrame 'df' has been defined. It outlines the process of converting data to Parquet for performance benefits.

```python
# Convert CSV to Parquet for faster querying and loading/restoring.

# This examples presumes you've run the above relevant polars cells (and have
# `df` defined as a polars dataframe. If you've imported and used pandas
# instead, this cell may not work at all or may not work as expected).

# Define the directory for the Parquet files.
this_dataset_parquet_directory = this_dataset_directory / "parquet"
if not this_dataset_parquet_directory.exists():
    print(f"Creating dataset parquet directory {this_dataset_parquet_directory}...")
    this_dataset_parquet_directory.mkdir(parents=True)

# NOTE: If you'd like, you can filter the `df` before dumping it. This can be a
# helpful strategy if you want to exclude data with missing values relevant to
# your research or more granularly filter the data than Dewey's web app allows
# for.
#
# This example filters out rows where "RAW_VISITOR_COUNTS" is null or the empty
# string.
#
# Other Important NOTE: If you want to do this filtering, make sure to replace
# `df` below with `filtered_df` in places where you want the filtered data
# instead of the entire data
```

--------------------------------

### Configure Direct Jupyter Download

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Enables and configures the direct download process for datasets within a Jupyter environment.

```python
# If you really want to do this, change this value to `True`.
DO_JUPYTER_DIRECT_DOWNLOAD: bool = False

if DO_JUPYTER_DIRECT_DOWNLOAD:
    import os
    from pathlib import Path

    from deweypy.auth import set_api_key
    from deweypy.downloads import DatasetDownloader, set_download_directory

    set_api_key("TODO-PUT-YOUR-API-KEY-HERE")
    # TODO: If you want, change the download directory.
    download_directory = Path(f".{os.sep}dewey-downloads")
    if not download_directory.exists():
        print(f"Creating download directory {download_directory}...")
        download_directory.mkdir(parents=True)
    set_download_directory(download_directory)

    # TODO (Dewey Team): Replace this with `api_url` without splitting or
    # whatever should be the main way to do this once it's all ready.
    downloader = DatasetDownloader(api_url.rsplit("/", 1)[-1])
    downloader.download()
```

--------------------------------

### Create DuckDB View from Parquet Files

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Define a SQL view in DuckDB that reads data from a specified directory of Parquet files. Adjust the path and date filter as needed for your data partitioning.

```python
# We can define a view to work with the data
con.sql("DROP VIEW mydata")  # drop if exists
con.sql(
    f"CREATE VIEW mydata AS SELECT * FROM '{parquet_directory}/*/*/*.parquet' WHERE DATE_RANGE_START < '2021-01-01'"
)  # Adjust * based on paritions
```

--------------------------------

### Query Data from DuckDB View

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Query the previously created view as if it were a regular table. This leverages the Parquet files for efficient data retrieval.

```python
# We can query the view like a table and it will use the parquet files for efficient querying
con.sql(
    "SELECT BRANDS, CITY, VISITS, DATE_RANGE_START FROM mydata WHERE CITY = 'Anchorage' AND visits > 1000 LIMIT 10"
)
```

--------------------------------

### Compare CSV and Parquet Scan Performance

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Prints the elapsed time for scanning CSV and Parquet files and calculates the speedup factor. This helps quantify the performance benefits of using Parquet.

```python
print("\nPerformance comparison:")
print(f"CSV scan time: {time_original_elapsed:.2f} seconds")
print(f"Parquet scan time: {time_parquet_elapsed:.2f} seconds")
print(f"Speedup: {time_original_elapsed / time_parquet_elapsed:.1f}x faster")
```

--------------------------------

### Analyze Parquet Data with DuckDB

Source: https://context7.com/dewey-data/deweypy/llms.txt

Connects to a DuckDB database to perform SQL queries on Parquet files via views.

```python
import duckdb
from pathlib import Path

parquet_dir = Path("./dewey-downloads/my-dataset/parquet")

# Connect to DuckDB (creates file if doesn't exist)
con = duckdb.connect("mydata.duckdb")

# Create a view over Parquet files
con.sql(f"""
    CREATE OR REPLACE VIEW mydata AS
    SELECT * FROM '{parquet_dir}/**/*.parquet'
    WHERE DATE_RANGE_START < '2021-01-01'
""")

# Query with SQL
result = con.sql("""
    SELECT
        BRANDS,
        CITY,
        VISITS,
        DATE_RANGE_START
    FROM mydata
    WHERE CITY = 'Anchorage' AND VISITS > 1000
    LIMIT 10
""")
print(result)

# Aggregate analysis
monthly_visits = con.sql("""
    SELECT
        YEAR(DATE_RANGE_START::DATE) AS year,
        MONTH(DATE_RANGE_START::DATE) AS month,
        SUM(VISITS) AS total_visits
    FROM mydata
    WHERE CITY = 'Anchorage'
    GROUP BY year, month
    ORDER BY year, month
""")
print(monthly_visits)

con.close()
```

--------------------------------

### Analyze Monthly Visit Trends in DuckDB

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Aggregate and analyze monthly visit trends for a specific city from the view. This query extracts year, month, and total visits, grouping and ordering the results.

```python
visits_anchor = con.sql(
    "SELECT YEAR(DATE_RANGE_START::DATE) AS year, MONTH(DATE_RANGE_START::DATE) AS month, SUM(VISITS) AS total_visits FROM mydata WHERE CITY = 'Anchorage' GROUP BY year, month ORDER BY year, month;"
)
visits_anchor
```

--------------------------------

### Filter and Partition DataFrame for Parquet

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Filters a DataFrame to include only rows with non-null and non-empty 'RAW_VISITOR_COUNTS', then adds a 'year_month' column by converting 'DATE_RANGE_START' to datetime and formatting it. Finally, it partitions the DataFrame by 'REGION' and 'year_month' and saves it as compressed Parquet files.

```python
filtered_df = df.filter(
    (pl.col("RAW_VISITOR_COUNTS").is_not_null()) & (pl.col("RAW_VISITOR_COUNTS") != "")
)
df_for_partitioning = df.with_columns(
    pl.col("DATE_RANGE_START")
    .str.to_datetime("%Y-%m-%d %H:%M:%S%.f", strict=True)
    .dt.strftime("%Y-%m")
    .alias("year_month")
)
partition_by = pl.PartitionByKey(
    base_path=this_dataset_parquet_directory,
    by=["REGION", "year_month"],
)
df_for_partitioning.sink_parquet(
    partition_by,
    compression="zstd",
    maintain_order=False,
    mkdir=True,
    lazy=False,
)
print(f"Saved partitioned files: {this_dataset_parquet_directory}")
```

--------------------------------

### Perform Advanced Analytics and Transformations

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Cast string columns to appropriate types and identify top locations by visitor count using aggregation and sorting.

```python
# Some more advanced analytics examples:

# Convert string columns to more specific types where needed.
df = df.with_columns(
    [
        pl.col("RAW_VISITOR_COUNTS").cast(pl.Int64, strict=False).alias("visitors"),
        pl.col("RAW_VISIT_COUNTS").cast(pl.Int64, strict=False).alias("visits"),
        pl.col("MEDIAN_DWELL").cast(pl.Float64, strict=False).alias("dwell_minutes"),
        pl.col("WKT_AREA_SQ_METERS").cast(pl.Float64, strict=False).alias("area_sq_m"),
        pl.col("DATE_RANGE_START")
        .str.to_datetime("%Y-%m-%d %H:%M:%S%.f")
        .dt.date()
        .alias("start_date"),
    ]
)

# Top N busiest locations by visitor count.
top_locations = (
    df.group_by(["PLACEKEY", "LOCATION_NAME", "CITY"])
    .agg(pl.col("visitors").sum().alias("total_visitors"))
    .sort("total_visitors", descending=True)
    .head(10)
    .collect()
)

print("Top Locations:")
print(top_locations)
```

--------------------------------

### Transform and Partition Data with Polars

Source: https://context7.com/dewey-data/deweypy/llms.txt

Converts CSV files to partitioned Parquet format to improve query performance and storage efficiency.

```python
# Load and transform data
df = pl.scan_csv(dataset_dir / "**/*.csv.gz")
df = df.with_columns(
    pl.col("DATE_RANGE_START")
        .str.to_datetime("%Y-%m-%d %H:%M:%S%.f", strict=True)
        .dt.strftime("%Y-%m")
        .alias("year_month")
)

# Save as partitioned Parquet (partition by region and month)
partition_by = pl.PartitionByKey(
    base_path=parquet_dir,
    by=["REGION", "year_month"],
)
df.sink_parquet(
    partition_by,
    compression="zstd",
    maintain_order=False,
    mkdir=True,
    lazy=False,
)

# Compare file sizes
csv_size = sum(f.stat().st_size for f in dataset_dir.glob("*.csv.gz"))
parquet_size = sum(f.stat().st_size for f in parquet_dir.glob("**/*.parquet"))
print(f"CSV size: {csv_size / (1024**3):.2f} GB")
print(f"Parquet size: {parquet_size / (1024**3):.2f} GB")
print(f"Compression ratio: {csv_size / parquet_size:.2f}x")

# Query Parquet files (typically 40x+ faster than CSV)
df_fast = pl.scan_parquet(parquet_dir / "**/*.parquet")
result = (
    df_fast.group_by("REGION")
    .agg(pl.col("visitors").sum())
    .collect()
)
```

--------------------------------

### Process Downloaded Data with Polars

Source: https://context7.com/dewey-data/deweypy/llms.txt

Uses lazy evaluation to scan, transform, and aggregate large datasets from gzipped CSV files.

```python
import polars as pl
from pathlib import Path

# Scan downloaded gzipped CSV files (lazy evaluation)
dataset_dir = Path("./dewey-downloads/my-dataset")
df = pl.scan_csv(dataset_dir / "**/*.csv.gz", low_memory=False)

# Get basic statistics
total_rows = df.select(pl.len()).collect()
print(f"Total rows: {total_rows.item():,}")

# View schema
schema = df.collect_schema()
for name, dtype in schema.items():
    print(f"  {name}: {dtype}")

# Transform and analyze
df = df.with_columns([
    pl.col("RAW_VISITOR_COUNTS").cast(pl.Int64, strict=False).alias("visitors"),
    pl.col("DATE_RANGE_START")
        .str.to_datetime("%Y-%m-%d %H:%M:%S%.f")
        .dt.date()
        .alias("start_date"),
])

# Aggregate by region and month
monthly_trends = (
    df.group_by([pl.col("start_date").dt.month().alias("month"), "REGION"])
    .agg(pl.col("visitors").sum().alias("monthly_visitors"))
    .sort(["REGION", "month"])
    .collect()
)
print(monthly_trends)
```

--------------------------------

### Scan CSV with Polars

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Initializes a lazy scan of CSV files using the Polars library.

```python
# NOTE: If you don't have `polars` installed, you should install it into your
# Python environment (I.E. with `pip`, `uv` or some other tool).
import polars as pl

# NOTE: You can set this to `True` if your machine doesn't have a lot of memory
# available.
polars_low_memory = False

# NOTE: Depending on the size of your dataset, this might take several seconds
# to several minutes or even longer in some cases.
df = pl.scan_csv(this_dataset_directory, low_memory=polars_low_memory)
```

--------------------------------

### Calculate Monthly Visitor Trends by State

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Groups data by month and region to sum monthly visitors, then sorts the results. Assumes 'df' is a pre-defined Polars DataFrame.

```python
monthly_trends = (
    df.group_by([pl.col("start_date").dt.month().alias("month"), "REGION"])
    .agg(pl.col("visitors").sum().alias("monthly_visitors"))
    .sort(["REGION", "month"])
    .collect()
)

print("Monthly Visitor Trends by State:")
print(monthly_trends)
```

--------------------------------

### Filter Partitioned Data

Source: https://github.com/dewey-data/deweypy/blob/main/README.md

Limit data processing by specifying partition boundaries using date keys.

```bash
--partition-key-before YYYY-MM-DD --partition-key-after YYYY-MM-DD
```

--------------------------------

### Inspect Polars Schema

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Retrieve and print the inferred schema of a Polars DataFrame to understand column names and data types.

```python
schema = df.collect_schema()
print("Column schema:")
for name, dtype in schema.items():
    print(f"  {name}: {dtype}")
```

--------------------------------

### Scan and Process Parquet Files with Polars

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Reads Parquet files, aggregates data by month and region, and measures the scan time. This snippet is used for performance comparison against CSV scanning.

```python
time_parquet_start = perf_counter()
df_pq = pl.scan_parquet(this_dataset_parquet_directory / "**/*.parquet")
monthly_trends_parquet = (
    df_pq.group_by([pl.col("start_date").dt.month().alias("month"), "REGION"])
    .agg(pl.col("visitors").sum().alias("monthly_visitors"))
    .sort(["REGION", "month"])
    .collect()
)
print("Parquet")
print(monthly_trends_parquet)
time_parquet_end = perf_counter()
time_parquet_elapsed = time_parquet_end - time_parquet_start
```

--------------------------------

### Aggregate Region Counts

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Group data by region and calculate counts for specific regions. Ensure the filter criteria match your specific dataset.

```python
# Region counts (specific to this dataset AK/WY example. Change this if you
# changed the dataset or filter).
region_counts = df.group_by("REGION").agg(pl.len().alias("count")).collect()
ak_count = region_counts.filter(pl.col("REGION") == "AK").select("count").item()
wy_count = region_counts.filter(pl.col("REGION") == "WY").select("count").item()
print(f"AK count: {ak_count:,}")
print(f"WY count: {wy_count:,}")
```

--------------------------------

### Analyze Polars Data

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Performs basic row count aggregation on the scanned Polars dataframe.

```python
# Very basic: Get a total row count directly from the `df`.
total_rows = df.select(pl.len()).collect()
print(f"Total rows: {total_rows.item():,}")
```

--------------------------------

### Scan and Process Gzipped CSVs with Polars

Source: https://github.com/dewey-data/deweypy/blob/main/notebook-examples/customized-monthly-patterns.ipynb

Reads gzipped CSV files, performs type casting and date parsing, and aggregates data by month and region. This snippet is useful for initial data loading and exploration from CSV sources.

```python
from time import perf_counter

time_original_start = perf_counter()
df_reset = pl.scan_csv(this_dataset_directory / "**/*.csv.gz", low_memory=False)
df_reset = df_reset.with_columns(
    [
        pl.col("RAW_VISITOR_COUNTS").cast(pl.Int64, strict=False).alias("visitors"),
        pl.col("RAW_VISIT_COUNTS").cast(pl.Int64, strict=False).alias("visits"),
        pl.col("MEDIAN_DWELL").cast(pl.Float64, strict=False).alias("dwell_minutes"),
        pl.col("WKT_AREA_SQ_METERS").cast(pl.Float64, strict=False).alias("area_sq_m"),
        pl.col("DATE_RANGE_START")
        .str.to_datetime("%Y-%m-%d %H:%M:%S%.f")
        .dt.date()
        .alias("start_date"),
    ]
)
monthly_trends_original = (
    df_reset.group_by([pl.col("start_date").dt.month().alias("month"), "REGION"])
    .agg(pl.col("visitors").sum().alias("monthly_visitors"))
    .sort(["REGION", "month"])
    .collect()
)
print("Original")
print(monthly_trends_original)
time_original_end = perf_counter()
time_original_elapsed = time_original_end - time_original_start
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.