### Bash: Install and Verify noaa_weather_hourly via pipx

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

This Bash script outlines the steps to install the noaa_weather_hourly tool as a standalone command-line application using pipx. It includes prerequisites like Python installation, pipx setup, the installation command, and verification steps using '--help'. It also shows example usage and upgrade/uninstall commands.

```bash
# Install Python 3.8 or newer (if not already installed)
# Windows: download from https://www.python.org/downloads/
# macOS: brew install python3
# Linux: sudo apt-get install python3

# Install pipx (Windows)
py -m pip install --user pipx
py -m pipx ensurepath

# Install pipx (Unix/macOS)
python3 -m pip install --user pipx
python3 -m pipx ensurepath

# Install noaa_weather_hourly
pipx install noaa_weather_hourly

# Verify installation
noaa_weather_hourly --help

# Expected output:
# usage: noaa_weather_hourly [-h] [-filename FILENAME] [-frequency FREQUENCY]
#                            [-max_records_to_interpolate MAX_RECORDS_TO_INTERPOLATE]
#
# noaa_weather_hourly - for processing raw NOAA LCD observed weather .csv files.
#
# optional arguments:
#   -h, --help            show this help message and exit
#   -filename FILENAME    File path to NOAA LCD CSV file to be processed
#   -frequency FREQUENCY  Time frequency of output CSV file
#   -max_records_to_interpolate MAX_RECORDS_TO_INTERPOLATE
#                         Maximum quantity of contiguous null records to be
#                         estimated using interpolation.

# Now available system-wide
cd ~/weather_projects/chicago
noaa_weather_hourly -filename "/path/to/LCD_USW00014939_2023.csv"

cd ~/weather_projects/minneapolis
noaa_weather_hourly -frequency 'D'

# Upgrade to latest version
pipx upgrade noaa_weather_hourly

# Uninstall
pipx uninstall noaa_weather_hourly
```

--------------------------------

### Install noaa_weather_hourly Package

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/README.md

Install the noaa_weather_hourly package globally using pipx after ensuring pipx is correctly installed and its path is set.

```bash
pipx install noaa_weather_hourly
```

--------------------------------

### Install pipx for Package Management

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/README.md

Install pipx, a utility for installing Python applications in isolated environments, making them available on your system's PATH. This is a prerequisite for installing the noaa_weather_hourly package.

```bash
py -m pip install --user pipx
py -m pipx ensurepath
```

```bash
python3 -m pip install --user pipx
python3 -m pipx ensurepath
```

--------------------------------

### Bash: Install noaa_weather_hourly as a Python Package

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

This Bash script details the process of installing the noaa_weather_hourly package from its source repository using pip. It covers cloning the repository, installing in development mode, installing from requirements, verifying the installation, and running the module from Python. It also shows the expected directory structure and package dependencies.

```bash
# Clone repository
git clone https://github.com/emskiphoto/noaa_weather_hourly.git
cd noaa_weather_hourly

# Install in development mode
pip install -e .

# Or install from requirements.txt
pip install -r requirements.txt

# Verify installation
python -m noaa_weather_hourly --help

# Run from Python code
python -m noaa_weather_hourly -filename "./data/3876540.csv" -frequency "D"

# Expected directory structure after cloning:
# noaa_weather_hourly/
# ├── noaa_weather_hourly/
# │   ├── __init__.py
# │   ├── __main__.py        # Main entry point
# │   ├── config.py          # Configuration constants
# │   ├── utils.py           # Helper functions
# │   └── data/
# │       └── isd-history.csv  # Station reference database
# ├── setup.py
# ├── pyproject.toml
# ├── requirements.txt
# └── README.md

# Package dependencies from pyproject.toml:
# - python >= 3.8, < 4.0
# - pandas > 1.3.5, <= 2.1.4
# - numpy > 1.14.6, <= 1.26.4
```

--------------------------------

### Get Current Working Directory (Example)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Retrieves the current working directory using the 'pathlib' library and displays it. This is used to determine the base directory for data file operations.

```python
dir_cwd = pathlib.Path.cwd()
dir_cwd
```

```text
Result:
WindowsPath('C:/Users/user/OneDrive/python_envs/noaa-weather-hourly-cli/dev')
```

--------------------------------

### Define Command Line Arguments (Example)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Illustrates the expected command-line arguments: 'filename' for specifying a CSV file and 'freqstr' for setting a frequency string. The default values are shown.

```python
filename, freqstr

```

```text
Result:
('', 'H')
```

--------------------------------

### Clone Repository with Git

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/README.md

Obtain the project code by cloning the GitHub repository using Git. This requires a local Git installation.

```bash
git clone https://github.com/emskiphoto/noaa_weather_hourly.git
```

--------------------------------

### Set Frequency String (Example)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Demonstrates how to set the 'freqstr' variable, which defines the time frequency for data processing. This can be adjusted based on specific data requirements.

```python
freqstr = '30T'
```

--------------------------------

### Configure NOAA LCD File Pattern Recognition (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Uses regular expressions to define and validate patterns for NOAA Local Climatological Data (LCD) files, supporting both v1 (numeric) and v2 (prefixed) formats. It includes examples of how to match filenames against these patterns.

```Python
from noaa_weather_hourly.config import (
    pattern_lcd1_input_file,
    pattern_lcd2_input_file,
    version_pattern_lcd_input,
    patterns_lcd_examples
)

# LCD v1 pattern: numeric filename like '3876540.csv'
print(f"LCD v1 pattern: {pattern_lcd1_input_file}")
print(f"LCD v1 example: {patterns_lcd_examples[0]}")

# LCD v2 pattern: format like 'LCD_USW00014939_2023.csv'
print(f"LCD v2 pattern: {pattern_lcd2_input_file}")
print(f"LCD v2 example: {patterns_lcd_examples[1]}")

# Example usage in file validation
import re

test_files = [
    "3876540.csv",                    # Valid v1
    "LCD_USW00014939_2023.csv",       # Valid v2
    "LCD_KMSP_2023.csv",              # Valid v2
    "weather_data.csv",               # Invalid
    "12345.csv",                      # Valid v1
]

for filename in test_files:
    v1_match = re.match(pattern_lcd1_input_file, filename)
    v2_match = re.match(pattern_lcd2_input_file, filename)

    if v1_match:
        print(f"{filename:30} -> LCD Version 1")
    elif v2_match:
        print(f"{filename:30} -> LCD Version 2")
    else:
        print(f"{filename:30} -> Not an LCD file")

# Access version-specific patterns programmatically
for version, pattern in version_pattern_lcd_input.items():
    print(f"Version {version}: {pattern}")
```

--------------------------------

### Generate Output File Name (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Constructs a formatted output file name using a template string 'file_output_format' and incorporates station name, start and end dates, and frequency string. It uses 'slugify' for station name formatting and assumes related variables are defined. It also includes a comment about potential OS limitations on file name length.

```python
# Save output file to current working directory (ie,
# where command line command was entered)
file_out_name = file_output_format.format(
            STATION_NAME = slugify(station_details['STATION NAME']),
            start_str = start_str,
            end_str = end_str,
            freqstr = freqstr)
file_out = dir_cwd / file_out_name
# what if file name is too long for current OS? - TO-DO
# file_out.as_posix().__len__()
file_out
```

--------------------------------

### Generate Date Range Strings

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Creates formatted string representations of the start and end datetimes from a DataFrame's index. These strings are typically used for logging or setting date boundaries.

```python
#     create timestamps for consolidated table df
start_dt = df.index[0]
end_dt = df.index[-1]
start_str = start_dt.strftime('%Y-%m-%d')
end_str = end_dt.strftime('%Y-%m-%d')
start_str, end_str
```

--------------------------------

### Determine Most Recent File and LCD Version

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This code determines the absolute most recent file from the identified version files and then determines its corresponding LCD version. It assumes versions start with '1' and adjusts the index accordingly.

```python
# 2. find the most recent file
file_last_modified = sorted([(f, f.stat().st_mtime) for
                  f in version_file_last_modified.values() if f != None],
           key=lambda x: x[1], reverse=True)[0][0]
# 3. Determine if most recent file is v1 or v2 format 'lcd_version'
# versions start with '1' so need to add 1 to zero-indexed list
lcd_version = list(version_file_last_modified.values())
                            .index(file_last_modified) + 1
# file_last_modified, lcd_version
```

--------------------------------

### Add Location Links to Station Details (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Demonstrates how to add a Google Maps URL to a dictionary of station details using latitude and longitude. This is useful for creating interactive links to station locations.

```Python
station_details = {
    'STATION NAME': 'CHICAGO OHARE INTERNATIONAL AIRPORT',
    'LAT': 41.978,
    'LON': -87.904,
    'ELEV(M)': 201.0
}

station_details['GOOGLE MAP'] = google_maps_url(
    station_details['LAT'],
    station_details['LON']
)

print("\nStation Details with Map Link:")
for key, value in station_details.items():
    print(f"{key:15} {value}")
```

--------------------------------

### Configure Processing Parameters for NOAA Data (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Sets default parameters for data processing, including the maximum acceptable percentage of null values for a timestamp, the maximum number of consecutive missing records to interpolate, and a dictionary of supported frequency string codes.

```Python
from noaa_weather_hourly.config import (
    pct_null_timestamp_max,
    max_records_to_interpolate,
    freqstr_frequency
)

# Maximum percentage of null values allowed per timestamp
print(f"Max null percentage per timestamp: {pct_null_timestamp_max * 100}%")
print("Timestamps exceeding this threshold are removed from dataset\n")

# Maximum consecutive missing values to interpolate
print(f"Max records to interpolate: {max_records_to_interpolate}")
print("Gaps larger than this preserve NaN values\n")

# Available frequency options and their descriptions
print("Supported frequency codes:")
for code, description in freqstr_frequency.items():
    print(f"  '{code:3}' - {description}")
```

--------------------------------

### Slugify String for Filenames (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Converts arbitrary strings into a filesystem-safe format by removing special characters, normalizing whitespace, and converting to lowercase. This is essential for creating valid filenames from station names or other potentially problematic identifiers. It handles various edge cases like apostrophes, parentheses, and multiple spaces.

```python
from noaa_weather_hourly.utils import slugify

# Convert station name to safe filename component
station_name = "CHICAGO O'HARE INTERNATIONAL"
safe_name = slugify(station_name)
print(f"Original: {station_name}")
print(f"Slugified: {safe_name}")

# Handle various problematic characters
test_cases = [
    "Minneapolis-St. Paul",
    "San Francisco (SFO)",
    "Hawai'i  Volcanoes",
    "Weather Station #42"
]

for name in test_cases:
    safe = slugify(name)
    print(f"{name:30} -> {safe}")

# Used in output filename generation
filename = f"{slugify(station_name)} 2020-01-01 to 2023-12-31 H.csv"
print(filename)
```

--------------------------------

### Import Project Modules

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Imports essential modules from the 'noaa_weather_hourly' package, specifically configuration settings and utility functions. These modules are required for the script's core functionalities.

```python
# import modules specific to this package 
from config import *
from utils import *
```

--------------------------------

### Python: Define and Print Custom Frequencies

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

This Python code snippet demonstrates how to define a list of custom time frequencies and then iterate through them to print each supported custom frequency. It's useful for verifying or showcasing the supported interval formats.

```python
# Build custom frequency strings
custom_frequencies = ['15T', '3H', '2D', '4W']
for freq in custom_frequencies:
    print(f"Custom frequency '{freq}' is supported")
```

--------------------------------

### Find and Sort Files by Modification Time (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Retrieves a list of all files within a directory that match a given regular expression pattern and sorts them by their last modification time. The sorting order can be ascending or descending. It returns a list of Path objects or an empty list if no files match.

```python
import pathlib
from noaa_weather_hourly.utils import find_files_re_pattern_sorted_last_modified

# Find all LCD v2 files for a specific station, sorted newest first
directory = pathlib.Path('/path/to/weather/data')
pattern = r'^LCD_USW00014939_[0-9]{4}.csv$'

files = find_files_re_pattern_sorted_last_modified(directory, pattern, descending=True)

if files:
    print(f"Found {len(files)} matching files:")
    for file in files:
        print(f"  - {file.name}")
else:
    print("No matching files found")

# Use ascending=False for oldest-first sorting
files_oldest_first = find_files_re_pattern_sorted_last_modified(
    directory, pattern, descending=False
)
```

--------------------------------

### Calculate Overall Statistical Comparison Metrics

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This commented-out Python code block suggests various methods for calculating an overall statistical comparison between pre-processed ('df_stats_pre') and post-processed ('df_stats_post') datasets. It includes calculations for mean differences and equality checks, intended for a high-level assessment of processing impact.

```python
# TO-DO:  Overall pre- post- statistics check
df_stats_pre.sub(df_stats_post).div(df_stats_pre).mean().mean()
# df_stats_pre.pct_change(df_stats_post)
# df_stats_pre.eq(df_stats_post).sum().sum()
# df_stats_pre.size
```

--------------------------------

### Generate Google Maps URL (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Creates a clickable Google Maps URL using provided latitude and longitude coordinates. This utility is useful for quickly visualizing the geographical location of weather stations or any point of interest directly on Google Maps. The generated URL includes the coordinates in a query parameter.

```python
from noaa_weather_hourly.utils import google_maps_url

# Generate map URL for weather station coordinates
lat = 41.978
lon = -87.904
map_url = google_maps_url(lat, lon)

print(f"Weather Station Location:")
print(f"Latitude: {lat}")
print(f"Longitude: {lon}")
print(f"Google Maps URL: {map_url}")
```

--------------------------------

### Custom Frequency Resampling

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Generates weather data at various time frequencies, including sub-hourly, daily, weekly, monthly, quarterly, and yearly intervals. The output file name reflects the specified frequency.

```APIDOC
## POST /process/resample

### Description
Processes NOAA LCD files and resamples the data to a specified time frequency. Supports a wide range of frequencies from minutely to yearly.

### Method
POST

### Endpoint
`/process/resample`

### Parameters
#### Query Parameters
- **filename** (string) - Optional - The path to the NOAA LCD CSV file to process. If not provided, the tool will auto-detect the most recent file(s).
- **frequency** (string) - Required - The desired output frequency. Supported codes include:
    - `'T'` for Minutely (e.g., `'15T'` for 15-minute intervals)
    - `'H'` for Hourly (e.g., `'3H'` for 3-hour intervals)
    - `'D'` for Daily
    - `'W'` for Weekly
    - `'M'` for Monthly
    - `'Q'` for Quarterly
    - `'Y'` for Yearly

### Request Example
```bash
# Generate 15-minute frequency output for a specific file
noaa_weather_hourly -frequency '15T' -filename './data/3876540.csv'

# Generate daily frequency using most recent file(s)
noaa_weather_hourly -frequency 'D'
```

### Response
#### Success Response (200)
- **output_path** (string) - The file path where the processed and resampled CSV is saved.
- **frequency** (string) - The frequency used for resampling.

#### Response Example
```json
{
  "output_path": "CHICAGO-OHARE-INTERNATIONAL 2020-01-01 to 2022-12-31 15T.csv",
  "frequency": "15T"
}
```
```

--------------------------------

### Resample NOAA Weather Data to Custom Frequencies

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Generates weather data outputs at various time frequencies, including sub-hourly, daily, weekly, monthly, quarterly, and yearly intervals. This allows for analysis at different granularities by specifying a frequency code. The output filename reflects the chosen frequency.

```bash
# Generate 15-minute frequency output
noaa_weather_hourly -frequency '15T' -filename './data/3876540.csv'

# Generate daily frequency using most recent file(s)
noaa_weather_hourly -frequency 'D'

# Generate 3-hour frequency output
noaa_weather_hourly -frequency '3H' -filename './data/3876540.csv'

# Generate weekly averages
noaa_weather_hourly -frequency 'W' -filename './data/3876540.csv'
```

--------------------------------

### Process Specific File

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Processes a single NOAA LCD CSV file and generates hourly weather data output. The tool automatically detects file versions, merges data, interpolates missing values, and resamples to hourly frequency by default.

```APIDOC
## POST /process/file

### Description
Processes a specific NOAA LCD CSV file and generates hourly weather data output. This endpoint handles data cleaning, merging, interpolation, and resampling.

### Method
POST

### Endpoint
`/process/file`

### Parameters
#### Query Parameters
- **filename** (string) - Required - The path to the NOAA LCD CSV file to process.

#### Request Body
(No specific request body defined for this CLI command, parameters are passed as flags/arguments)

### Request Example
```bash
noaa_weather_hourly -filename "./data/3876540.csv"
```

### Response
#### Success Response (200)
- **output_path** (string) - The file path where the processed CSV is saved.
- **station_info** (object) - Details about the weather station.
- **null_percentage_comparison** (object) - Comparison of null value percentages before and after processing.

#### Response Example
```json
{
  "output_path": "/path/to/output/MINNEAPOLIS-ST-PAUL-INTERNATIONAL-AIRPORT 2020-01-01 to 2023-12-31 H.csv",
  "station_info": {
    "USAF": "720340",
    "WBAN": "03876",
    "STATION NAME": "MINNEAPOLIS-ST PAUL INTERNATIONAL AIRPORT",
    "CTRY": "US",
    "STATE": "MN",
    "ICAO": "KMSP",
    "LAT": 44.883,
    "LON": -93.217,
    "ELEV(M)": 255.0
  },
  "null_percentage_comparison": [
    {
      "Column": "AltimeterSetting",
      "% NaN - Source": "0.00%",
      "% NaN - Processed": "0.00%",
      "Source Mean": 30.01,
      "Processed Mean": 30.01,
      "% Difference": "0.00%"
    }
  ]
}
```
```

--------------------------------

### Import Core Libraries and Configure Autocomplete

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Imports standard Python libraries such as pandas, csv, pathlib, and re. It also disables the Jedi autocompletion engine in the Jupyter notebook environment to resolve potential conflicts.

```python
import pandas as pd
import csv
import pathlib
import re
# turn off Jedi autocomplete (that was causing more problems than benefits post Win10 update 3-13-2020)
%config Completer.use_jedi = False
```

--------------------------------

### Add Package Path to PYTHONPATH

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Configures the Python environment by adding the script's package directory to the PYTHONPATH. This is crucial for the Jupyter notebook development version to load necessary source modules.

```python
# direct the jupyter notebook development script to add package path to PYTHONPATH
# to allow for loading of source modules.
import sys
sys.path.append('../noaa_weather_hourly')
```

--------------------------------

### Calculate and Format Percentage of Null Data (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Calculates the percentage of null (missing) values for each column in the DataFrame and formats the output for readability. It prepares a DataFrame showing 'Percent N/A' and then formats these percentages as strings with a '%' sign. It also renames columns by removing the 'Hourly' prefix for cleaner display.

```python
#     Check what percentage of data has null data and print to screen
df_pct_null_data = pd.DataFrame({'Percent N/A': df.isnull().sum().divide(len(df)).round(3)})
df_pct_null_data_pre_formatted = df_pct_null_data['Percent N/A']\
                            .apply(lambda n: '{:,.1%}'.format(n))
# remove 'Hourly' prefix for display only
col_rename_remove_hourly = {col_ : col_.replace('Hourly', '') for
                            col_ in df_pct_null_data_pre_formatted.index}
```

--------------------------------

### Process NOAA Weather Data with Custom Frequency

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/README.md

Generate weather data files with frequencies other than hourly using the -frequency argument. Supported frequencies include '15T' (15-minute), 'D' (Daily), 'W' (Weekly), 'M' (Monthly), 'Q' (Quarterly), and 'Y' (Yearly).

```bash
`noaa_weather_hourly -frequency '15T' '<path_to_LCD_CSV_file>'`
```

```bash
`noaa_weather_hourly -frequency 'D'`
```

--------------------------------

### Filter and Display Station Details (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Filters out specific 'BEGIN' and 'END' keys from a dictionary containing station details to avoid confusion. It then creates a new dictionary `station_details_display` with the remaining details and proceeds to print these details in a formatted manner, displaying the station's properties. This helps in presenting relevant metadata about the weather station.

```python
# exclude station lifetime history dates - could cause confusion
station_details_exclude = ['BEGIN', 'END']
station_details_display = {k_ : v_ for k_, v_ in station_details.items() if
                           k_ not in station_details_exclude}

print('--------------------------------------------')
print('------ ISD Weather Station Properties ------')
print('--------------------------------------------')
for k_, v_ in station_details_display.items():
    print("{:<15} {:<10}".format(k_, v_))
print('
')
```

--------------------------------

### Define Data Columns for NOAA Weather Processing (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Specifies the columns used and processed by the noaa_weather_hourly tool. It distinguishes between all processed columns, core weather data columns, special columns like sunrise/sunset, and metadata columns.

```Python
from noaa_weather_hourly.config import (
    cols_noaa_processed,
    cols_data,
    cols_sunrise_sunset,
    cols_date_station
)

# All columns processed from NOAA LCD files
print("Columns processed by noaa_weather_hourly:")
for col in cols_noaa_processed:
    print(f"  - {col}")

# Data columns only (excludes DATE and STATION)
print(f"\nWeather data columns: {len(cols_data)}")
print(cols_data)

# Special handling columns (sunrise/sunset times)
print(f"\nSunrise/Sunset columns: {cols_sunrise_sunset}")

# Metadata columns
print(f"\nMetadata columns: {cols_date_station}")

# Typical output file structure after processing
output_columns = [
    'AltimeterSetting',
    'DewPointTemperature',
    'DryBulbTemperature',
    'Precipitation',
    'PressureChange',
    'RelativeHumidity',
    'StationPressure',
    'Sunrise',
    'Sunset',
    'Visibility',
    'WetBulbTemperature',
    'WindDirection',
    'WindGustSpeed',
    'WindSpeed',
    'No source data'
]

print("\nOutput CSV columns (Hourly prefix removed):")
for col in output_columns:
    print(f"  - {col}")
```

--------------------------------

### Control Interpolation of Missing Weather Records

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Configures the maximum number of consecutive missing data records that the tool will interpolate. This provides granular control over the gap-filling process, ensuring that only short gaps are filled automatically, preventing potential inaccuracies from over-interpolation.

```bash
# Limit interpolation to maximum 12 consecutive missing hours
noaa_weather_hourly -max_records_to_interpolate 12 -filename './data/3876540.csv'

# Use default interpolation limit (24 hours) with custom frequency
noaa_weather_hourly -frequency 'D'
```

--------------------------------

### Compare Mean Values Before and After Processing

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This Python code compares the mean values of the source and processed datasets. It concatenates the mean statistics, calculates the percentage difference, and formats the output for clarity. The index is also renamed for better readability.

```python
df_mean_comp = pd.concat([df_stats_pre.loc['mean'].T, df_stats_post.loc['mean'].T],
                         axis=1, keys=['Source Mean', 'Processed Mean']).round(2)
df_mean_comp['% Difference'] = df_mean_comp.pct_change(axis=1).iloc[:,-1]\   .fillna(0).round(4)\   .apply(lambda n: '{:,.2%}'.format(n))
df_mean_comp.rename(index=col_rename_remove_hourly, inplace=True)
df_mean_comp
```

--------------------------------

### Display DataFrame Shape and Head (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This snippet demonstrates how to display the shape (number of rows and columns) and the first few rows of a Pandas DataFrame. It's useful for quickly understanding the dimensions and structure of the loaded data.

```python
df.shape
df.head()
```

--------------------------------

### Process Specific NOAA Weather File

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/README.md

Use the noaa_weather_hourly command-line tool to process a specific NOAA LCD .csv file. Provide the path to the file using the -filename argument.

```bash
`noaa_weather_hourly -filename ".\data\3876540.csv"`
```

--------------------------------

### Combine and Display Null Data Comparison (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Concatenates pre-formatted and post-formatted dataframes containing null percentages and renames the index. This is useful for comparing null data before and after processing. Requires pre-formatted and post-formatted dataframes.

```python
df_pct_null_comp = pd.concat([df_pct_null_data_pre_formatted, 
                             df_pct_null_data_post_formatted], 
                            axis=1).rename(index=col_rename_remove_hourly)
df_pct_null_comp
```

--------------------------------

### Control Interpolation Limits

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Allows users to configure the maximum number of consecutive missing values that the tool will interpolate. This provides granular control over gap-filling behavior in the processed data.

```APIDOC
## POST /process/interpolate

### Description
Processes NOAA LCD files while controlling the maximum number of consecutive missing records that will be interpolated. This helps manage data imputation based on the expected data gap.

### Method
POST

### Endpoint
`/process/interpolate`

### Parameters
#### Query Parameters
- **filename** (string) - Optional - The path to the NOAA LCD CSV file to process. If not provided, the tool will auto-detect the most recent file(s).
- **max_records_to_interpolate** (integer) - Optional - The maximum number of consecutive missing records to interpolate. Defaults to 24.
- **frequency** (string) - Optional - The desired output frequency (e.g., 'H', 'D', '15T'). Used in conjunction with interpolation.

### Request Example
```bash
# Limit interpolation to maximum 12 consecutive missing hours for a specific file
noaa_weather_hourly -max_records_to_interpolate 12 -filename './data/3876540.csv'

# Use default interpolation limit (24 hours) with daily frequency
noaa_weather_hourly -frequency 'D'
```

### Response
#### Success Response (200)
- **output_path** (string) - The file path where the processed CSV is saved.
- **interpolation_limit** (integer) - The maximum number of records interpolated.

#### Response Example
```json
{
  "output_path": "MINNEAPOLIS-ST-PAUL-INTERNATIONAL-AIRPORT 2020-01-01 to 2023-12-31 H.csv",
  "interpolation_limit": 12
}
```
```

--------------------------------

### Format LCD Input Filenames as String

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This code converts a list of file objects (files_lcd_input) into a multi-line string, where each line is the name of a file. This is useful for display or logging purposes.

```python
# string version of filenames from files_lcd_input as vertical list
files_lcd_input_names_str = "\n".join([f_.name for f_ in files_lcd_input])
# files_lcd_input_names_str
```

--------------------------------

### Calculate and Display Percentage of Null Data in DataFrame (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Calculates the percentage of null (N/A) values for each column in a DataFrame and formats the output for display. It uses pandas for data manipulation.

```python
df_pct_null_data_post = pd.DataFrame({'Percent N/A': df_out.isnull()\
                              .sum().divide(len(df_out)).round(4)})
df_pct_null_data_post_formatted = df_pct_null_data_post['Percent N/A']\
                                .apply(lambda n: '{:,.2%}'.format(n))
display(df_pct_null_data_post_formatted.rename(
                index = col_rename_remove_hourly))
```

--------------------------------

### Execute Optional Frequency Resampling

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This conditional Python code executes the resampling of 'df_out' based on the 'freqstr'. If 'run_resample' is true and 'resample_interpolate' is true, it resamples using interpolation; otherwise, it resamples using the mean. This allows for adjusting the data's time frequency.

```python
if run_resample:
#     resample via interpolation
    if resample_interpolate:
        df_out = df_out.resample(freqstr).interpolate()
# or resample using mean
    elif not resample_interpolate:
        df_out = df_out.resample(freqstr).mean()
```

--------------------------------

### Find Latest File in Directory (Python)

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Locates the most recently modified file within a specified directory that matches a given regular expression pattern. It utilizes pathlib for path manipulation and standard regex for pattern matching. Returns a Path object for the latest file or None if no matches are found.

```python
import pathlib
from noaa_weather_hourly.utils import find_latest_file

# Find most recent file matching pattern in data directory
directory = pathlib.Path('/path/to/weather/data')
pattern = r'LCD_USW00014939_[0-9]{4}.csv'

latest_file = find_latest_file(directory, pattern)

if latest_file:
    print(f"Most recent file: {latest_file.name}")
    print(f"Full path: {latest_file.as_posix()}")
    print(f"Last modified: {latest_file.stat().st_mtime}")
else:
    print("No matching files found")
```

--------------------------------

### Set Development Mode Flag

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

A boolean flag to enable or disable development-specific features, such as visualizations and intermittent quality checks. This setting is stored in the 'config.py' file.

```python
# turn on development mode to enable visualizations and other intermittent
# quality checks that aren't avaiable in the final version
is_development = True
# is_development = False
```

--------------------------------

### Define Data Directory Based on Development Mode

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Sets the 'dir_data' variable, which points to the directory containing the CSV weather files. It uses the current working directory in production and a subdirectory named 'data' in development mode. An assertion verifies that the directory exists.

```python
# using 'data' within 'dev' folder - potentially different from 'data' in package folder
# dir_data should be dir_cwd in production script
if is_development:
    dir_data = dir_cwd / 'data'
else:
    dir_data = dir_cwd
assert dir_data.is_dir()
```

--------------------------------

### Auto-detect and Process Most Recent NOAA LCD Files

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Automatically identifies and processes the most recently modified NOAA LCD files in the current directory. The tool groups files belonging to the same weather station, merges them, and generates a single output CSV. This is useful for updating datasets with the latest available weather information.

```bash
# Navigate to directory containing LCD files
cd /path/to/weather/data

# Process most recent file(s) automatically
noaa_weather_hourly
```

--------------------------------

### Auto-detect and Process Recent Files

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Automatically detects and processes the most recently modified NOAA LCD file(s) in the current directory. It groups files from the same station ID, merges them, and outputs a single, analysis-ready CSV file.

```APIDOC
## POST /process/recent

### Description
Automatically scans the current directory for NOAA LCD files (v1 and v2), identifies the most recent ones, groups them by station ID, merges, and processes them into a single output CSV file.

### Method
POST

### Endpoint
`/process/recent`

### Parameters
(No specific parameters, operates on the current directory)

### Request Example
```bash
cd /path/to/weather/data
noaa_weather_hourly
```

### Response
#### Success Response (200)
- **output_path** (string) - The file path where the processed CSV is saved.
- **processed_files** (array) - List of files that were processed.

#### Response Example
```json
{
  "output_path": "CHICAGO-OHARE-INTERNATIONAL 2020-01-01 to 2022-12-31 H.csv",
  "processed_files": [
    "LCD_USW00014939_2020.csv",
    "LCD_USW00014939_2021.csv",
    "LCD_USW00014939_2022.csv"
  ]
}
```
```

--------------------------------

### Format WBAN Index to 5-Character String

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Ensures the WBAN index in the DataFrame is a 5-character string by zero-padding it. This is crucial for consistent indexing and lookups. It then samples and sorts the DataFrame to display a preview.

```python
df_isd_history.index.index = df_isd_history.index.str.zfill(5)
df_isd_history.sample(5).sort_index()
```

--------------------------------

### Join DataFrames for Combined Comparison (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Joins the 'df_pct_null_comp' DataFrame with the 'df_mean_comp' DataFrame. This operation combines null percentage comparisons with mean comparisons, likely for a comprehensive data analysis report. It assumes both DataFrames have compatible indices for joining.

```python
df_pct_null_comp.join(df_mean_comp)
```

--------------------------------

### Find and Sort LCD Input Files by Date

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Identifies all files matching version 1 or version 2 LCD naming patterns within the data directory. It then sorts these files by their last modified date in descending order using a helper function.

```python
# 1. find all files that match v1 or v2 naming and sort by last modified date descending
# create 'version_files' dictionary with LCD version number as key and list of matching 
# files as values
version_files = {v_ : find_files_re_pattern_sorted_last_modified(dir_data, pattern_) for
                 v_, pattern_ in version_pattern_lcd_input.items()}
version_files
```

```text
Result:
{1: [WindowsPath('C:/Users/user/OneDrive/python_envs/noaa-weather-hourly-cli/dev/data/3876540.csv'),
  WindowsPath('C:/Users/user/OneDrive/python_envs/noaa-weather-hourly-cli/dev/data/3875753.csv')],
 2: []}
```

--------------------------------

### List CSV Files in Data Directory

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Generates a sorted list of all files with the '.csv' extension present in the 'dir_data' directory. This is used to identify available weather data files.

```python
# pretty name of directory
dir_data_posix = dir_data.as_posix()
# list of all .csv files in dir_data
dir_data_csv_files = sorted([f_.name for f_ in dir_data.glob('*.csv') if f_.is_file()])
```

--------------------------------

### Add Google Maps URL to Station Details

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

If latitude and longitude are available in `station_details`, this script constructs a Google Maps search URL and adds it to the `station_details` dictionary under the key 'GOOGLE MAP'. The original DataFrame `df_isd_history` is then deleted.

```python
if station_details['LAT'] != 'Unknown':
     # add url to google maps search of lat, lon values to station_details
    google_maps_lat_lon_url = """https://maps.google.com/?q={lat},{long}"""
    google_maps_url = google_maps_lat_lon_url.format(lat = station_details['LAT'],
                                                    long = station_details['LON'])
    station_details['GOOGLE MAP'] = google_maps_url

# delete df_isd_history - no longer needed
del df_isd_history
```

--------------------------------

### Process Specific NOAA LCD File to Hourly Data

Source: https://context7.com/emskiphoto/noaa_weather_hourly/llms.txt

Processes a single NOAA Local Climatological Data (LCD) CSV file and outputs hourly weather data. It cleans, merges, and interpolates data, saving the result to a uniquely named CSV file. This is the basic usage for processing a known file.

```bash
# Process a specific v1 LCD file with default hourly frequency
noaa_weather_hourly -filename "./data/3876540.csv"
```

--------------------------------

### Create Initial Output DataFrame and Ensure Hourly Frequency

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This Python code concatenates resampled series (from the 'dfs' dictionary) into a single DataFrame, removes duplicates, and sets the index to hourly frequency. It also enforces a 'float' data type to prevent errors in subsequent interpolation steps. The original DataFrames 'dfs' and 'df' are deleted to free up memory.

```python
# join the resampled series from the previous step, remove any
# duplicates and ensure the index is recognized as hourly frequency.
# df_out = pd.concat(dfs, axis=1).drop_duplicates().asfreq('H')
# important to enforce dtype 'float' as 'HourlyRelativeHumidity' and 
# other columns had a 'Float64' (capital 'F') that generated
# errors in interpolation step.
df_out = pd.concat(dfs, axis=1).drop_duplicates() \
                .asfreq('H').astype(float)
del dfs
del df
df_out
```

--------------------------------

### Display Mean Comparison DataFrame (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Displays a DataFrame named 'df_mean_comp', presumably containing mean comparison statistics. No specific logic is shown, only the display of the existing DataFrame.

```python
df_mean_comp
```

--------------------------------

### Retrieve Latest Station Details

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Retrieves the latest available station details from `df_isd_history`. It prioritizes WBAN lookup. If not found by WBAN, it looks up by CALL sign and selects the entry with the latest 'END' date. If neither is found, it returns 'Unknown' for all fields.

```python
if station_details_available_wban:
    station_details = dict(df_isd_history.loc[station_wban].reset_index()
                       .sort_values('END', ascending=False).iloc[0])
elif station_details_available_call:
    station_details = dict(df_isd_history.loc[
                        df_isd_history['CALL'] == station_call]
                       .reset_index().sort_values('END',
                          ascending=False).iloc[0])
else:
    station_details = {col_ : 'Unknown' for col_ in df_isd_history.columns}
```

--------------------------------

### Display DataFrame Information

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Prints a concise summary of the `df_isd_history` DataFrame, including the index dtype and columns, non-null values and their dtypes, and memory usage. Useful for understanding the structure and content of the DataFrame.

```python
df_isd_history.info()
```

--------------------------------

### Create DataFrame from NOAA Hourly Data (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This snippet reads multiple CSV files, specified by the 'files_usecols' dictionary, using pandas. It selects only the columns defined in 'cols_' for each file, parses the 'DATE' column as datetime objects, and sets it as the index. The data from all files is concatenated, duplicates are removed, and the index is reset and sorted by date. This function is crucial for consolidating diverse hourly weather observations into a usable format.

```python
import pandas as pd

# Assuming files_usecols is a dictionary where keys are file paths and values are lists of column names to use
# Example: files_usecols = {"file1.csv": ["DATE", "STATION", "HourlyDryBulbTemperature"], "file2.csv": ["DATE", "STATION", "HourlyDryBulbTemperature"]}
def create_dataframe_from_files(files_usecols):
    df = pd.concat((pd.read_csv(f_, usecols=cols_, parse_dates=['DATE'],
                                index_col='DATE', low_memory=False) for
                    f_, cols_ in files_usecols.items()), axis=0)
                    .reset_index().drop_duplicates()
    df = df.set_index('DATE', drop=True).sort_index()
    return df

# Example usage:
# df = create_dataframe_from_files(files_usecols)
# print(df.head())
```

--------------------------------

### Load ISD History Station Details into DataFrame (Python)

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This code reads the 'isd-history.csv' file into a Pandas DataFrame, using 'WBAN' as the index and specifying it as an object type to preserve leading zeros. The DataFrame is then sorted by 'USAF' and 'BEGIN' date in descending order for efficient lookup. This DataFrame contains historical details for weather stations.

```python
df_isd_history = pd.read_csv(file_isd_history, index_col='WBAN',
                     dtype={'WBAN': object}).sort_values(
                    by=['USAF', 'BEGIN'], ascending=[True, False])
```

--------------------------------

### Display Pre-Processing Null Value Statistics

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This code snippet displays a formatted table of the percentage of missing values per column before any data processing. It uses pandas DataFrames and the display function, likely within an environment like Jupyter Notebook.

```python
print('--------------------------------------------------------------')
print('------ Percent Null Values by Column Before Processing -------')
print('--------------------------------------------------------------')
display(df_pct_null_data_pre_formatted.rename(index=col_rename_remove_hourly))
```

--------------------------------

### Generate Post-Processing Statistical Comparison

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This Python code calculates and displays descriptive statistics for the post-processed DataFrame 'df_out', using the same columns as the pre-processed statistics DataFrame 'df_stats_pre'. This facilitates a comparison of the data's statistical properties before and after the cleaning and resampling operations.

```python
# create general statistics on post-processed dataset for
# comparison with pre-processed dataset to understand how/if 
# processing significantly altered series values
df_stats_post = df_out[df_stats_pre.columns].describe()
df_stats_post
```

--------------------------------

### Extract File Headers

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This snippet reads the header (fieldnames) from each file in the 'files_lcd_input' list. It prioritizes using csv.DictReader for efficiency over pandas for just reading headers, and sorts the fieldnames for consistency.

```python
# read only headers of each file in files_lcd_input
files_columns = {}
for file_ in files_lcd_input:
    try:
#         this is 30x faster than pd.read_csv(file_, index_col=0, nrows=0).columns.tolist()
        with open(file_, 'r') as infile:
            reader = csv.DictReader(infile)
            fieldnames = reader.fieldnames
        files_columns[file_] = sorted(fieldnames)
    except:
        continue
```

--------------------------------

### Retrieve Station Names as NumPy Array

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

Extracts all station names from the 'STATION NAME' column of the `df_isd_history` DataFrame and returns them as a NumPy array. This is useful for searching or listing station names.

```python
df_isd_history['STATION NAME'].values
```

--------------------------------

### Determine Resampling Logic Based on Frequency

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This Python code determines whether to perform resampling and if interpolation is needed based on the input frequency string ('freqstr'). It calculates the frequency delta and compares it to the DataFrame's index frequency delta to decide on the resampling strategy (interpolation for higher frequency, mean for lower).

```python
# freqstr = '30T'

run_resample = False
resample_interpolate = False
#     what is the delta value of the input freqstr?
freqstr_delta = pd.date_range(start_dt, periods=100,
                           freq=freqstr).freq.delta
# If the freqstr is not 'H', run the resample process
if freqstr != 'H':
    run_resample = True

# If the input freqstr is higher frequency 
# than df_out, resample using interpolation
resample_interpolate = freqstr_delta < df_out.index.freq.delta
```

--------------------------------

### Display DataFrame Information After Processing

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This simple Python command displays concise summary information about the 'df_out' DataFrame after all processing steps have been completed. This includes the index type, column data types, and memory usage.

```python
df_out.info()
```

--------------------------------

### Process Most Recent NOAA Weather Files

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/README.md

Automatically process the newest weather files in the current directory. The tool selects the latest files based on modification date and groups data from the same weather station.

```bash
`noaa_weather_hourly`
```

--------------------------------

### Resample and Interpolate Individual Columns

Source: https://github.com/emskiphoto/noaa_weather_hourly/blob/main/dev/noaa-weather-hourly.ipynb

This Python code iterates through each column of a DataFrame, drops NaN values, and then resamples the data to an hourly frequency using the mean. It stores the resampled series in a dictionary. This is intended to create complete datetime indexes for each column, with remaining NaNs to be handled by interpolation later.

```python
%%time
dfs = {}
# individually resample each column on an hourly frequency.
# This will produce series
# with perfect, complete datetime indexes.  However, it is quite
# possible that NaN values will remain (ie. a contiguous 3-hour
# period of NaN values).  Remaining NaN values will
# be resolved through interpolation later in the script.
# this method is used because NaN values can appear at different timestamps
# in each column
# at this point the df should contain only numeric data
for col_ in df.columns:
    print(col_)
    dfs[col_] = df[col_].dropna().resample('H').mean()
```