### Install project skills Source: https://github.com/catalyst-cooperative/pudl/blob/main/CLAUDE.md Command to install skills defined in skills-lock.json. ```bash pixi run install-skills ``` -------------------------------- ### Initialize Worktree Environment Source: https://github.com/catalyst-cooperative/pudl/blob/main/AGENTS.md Run this command to install dependencies and prepare the environment for a new worktree. ```bash pixi install && pixi run prek-install ``` -------------------------------- ### Launch Dagster UI Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/run_the_etl.rst Start the Dagster webserver after configuring the environment. ```console $ pixi run dg dev ``` -------------------------------- ### Install PUDL Development Environment Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/dev_setup.rst Installs all project dependencies and the PUDL package in editable mode using pixi. ```console $ pixi install ``` -------------------------------- ### Datastore Directory Structure Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/datastore.rst Example of the local file hierarchy created under the $PUDL_INPUT directory. ```text data/censusdp1tract/ data/eia860/ data/eia860m/ data/eia861/ data/eia923/ data/epacems/ data/ferc1/ data/ferc2/ data/ferc60/ data/ferc714/ data/phmsagas/ ``` -------------------------------- ### API Request via URL Parameters Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Example of a GET request with all parameters included directly in the URL string. ```http http://api.eia.gov/v2/electricity/retail-sales/data/?api_key=xxxxxx&facets[stateid][]=CO&facets[sectorid][]=RES&frequency=monthly ``` -------------------------------- ### Configure Notebook Environment Source: https://github.com/catalyst-cooperative/pudl/blob/main/notebooks/work-in-progress/sec10k_irregularities_figures.ipynb Enable autoreload and install necessary dependencies for data processing and visualization. ```python %load_ext autoreload %autoreload 3 ``` ```python %pip install matplotx ``` ```python %pip install polars ``` -------------------------------- ### Verify Dagster project installation Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/dev_setup.rst Run this command to confirm the Dagster project loads correctly after environment installation. ```console $ pixi run dg check defs --verbose ``` -------------------------------- ### Install dependencies Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/debug-column-mapping.ipynb Optional installation command for xlrd, which may be required for specific extractor checks. ```python # this is occasionally required for running the extractor check down below. # ! pip install xlrd ``` -------------------------------- ### Response with Multiple Data Columns Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Example response containing both price and revenue data. ```json data: [ { period: "2010", stateid: "AZ", stateDescription: "Arizona", sectorid: "TRA", sectorName: "transportation", price: "0", revenue: "0", price-units: "cents per kilowatthour", revenue-units: "million dollars" }, … ] ``` -------------------------------- ### Data Response with Price Column Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Example response containing the requested price data and units. ```json response: { total: "7440", dateFormat: "YYYY", frequency: "annual", data: [ { period: "2010", stateid: "AZ", stateDescription: "Arizona", sectorid: "TRA", sectorName: "transportation", price: "0", price-units: "cents per kilowatthour" }, { period: "2010", stateid: "AR", stateDescription: "Arkansas", sectorid: "ALL", sectorName: "all sectors", price: "7.28", price-units: "cents per kilowatthour" }, … //additional returns ] } ``` -------------------------------- ### Array Syntax Response Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Example response confirming that array syntax produces the same output as indexed parameters. ```json data: [ { period: "2010", stateid: "AZ", stateDescription: "Arizona", sectorid: "TRA", sectorName: "transportation", price: "0", revenue: "0", price-units: "cents per kilowatthour", revenue-units: "million dollars" }, … ] ``` -------------------------------- ### Initialize Database Schema Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/run_the_etl.rst Run this command to create the database with the schema defined in the codebase. ```bash pixi run alembic upgrade head ``` -------------------------------- ### Build documentation with Pixi Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/build_docs.rst Regenerates all documentation from scratch by removing previous outputs. ```console $ pixi run docs-build ``` -------------------------------- ### Filter by Start Date Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Constrains the API response to data points occurring after the specified start date. ```http https://api.eia.gov/v2/electricity/retail-sales/data?api_key=xxxxxx&data[]=price&facets[sectorid][]=RES&facets[stateid][]=CO&frequency=monthly&start=2008-01-31 ``` -------------------------------- ### Initialize Environment and Imports Source: https://github.com/catalyst-cooperative/pudl/blob/main/notebooks/work-in-progress/eia191-annual-vs-monthly.ipynb Sets up autoreload, logging, and imports necessary libraries for data manipulation and visualization. ```python %load_ext autoreload %autoreload 2 import io import logging import zipfile from pathlib import Path import pandas as pd import requests import matplotlib.pyplot as plt logging.basicConfig(level=logging.INFO, format="%(message)s") logger = logging.getLogger(__name__) pd.options.display.max_columns = None pd.options.display.max_rows = 50 ``` -------------------------------- ### Initialize logger Source: https://github.com/catalyst-cooperative/pudl/blob/main/notebooks/work-in-progress/explore_fuel_costs_eiaapi.ipynb Sets up a logger instance for tracking execution progress. ```python logger = pudl.logging_helpers.get_logger(__name__) ``` -------------------------------- ### Install Dagster Agent Skills Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/run_the_etl.rst Install the configured Dagster agent skills using the project's pixi task. ```console $ pixi run install-skills ``` -------------------------------- ### Initialize dataset paths Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/debug-column-mapping.ipynb Set up paths for raw data and mapping CSV files based on the specified dataset. ```python dataset = "eia861" doi_path = getattr(ZenodoDoiSettings(), dataset).replace("/", "-") pudl_paths = pudl.workspace.setup.PudlPaths() data_path = os.path.join(pudl_paths.pudl_input,dataset,doi_path) # Get path to raw data map_path = os.path.join(Path(pudl.package_data.__file__).parents[0], dataset) # Get path to mapping CSVs ds = pudl.workspace.datastore.Datastore(pudl_paths.pudl_input) ``` -------------------------------- ### Filter by Start and End Date Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Constrains the API response to a specific time span using both start and end parameters. ```http https://api.eia.gov/v2/electricity/retail-sales/data?api_key=xxxxxx&data[]=price&facets[sectorid][]=RES&facets[stateid][]=CO&frequency=monthly&sta rt=2008-01-31&end=2008-03-01 ``` -------------------------------- ### Row Count Diff Example: Unexpected Changes Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/data_validation_reference.rst Example diff showing unexpected row count changes that require investigation. ```diff diff --git a/dbt/seeds/etl_full_row_counts.csv b/dbt/seeds/etl_full_row_counts.csv index d9a5f0ec7..2b40f3ad7 100644 --- a/dbt/seeds/etl_full_row_counts.csv +++ b/dbt/seeds/etl_full_row_counts.csv @@ -3318,7 +3318,7 @@ out_ferc1__yearly_steam_plants_fuel_sched402,2020,1250 out_ferc1__yearly_steam_plants_fuel_sched402,2021,1152 out_ferc1__yearly_steam_plants_fuel_sched402,2022,1196 -out_ferc1__yearly_steam_plants_fuel_sched402,2023,1210 +out_ferc1__yearly_steam_plants_fuel_sched402,2023,1215 -out_ferc1__yearly_steam_plants_fuel_sched402,2024,1224 +out_ferc1__yearly_steam_plants_fuel_sched402,2024,1221 out_ferc1__yearly_steam_plants_sched402,1994,1411 out_ferc1__yearly_steam_plants_sched402,1995,1448 out_ferc1__yearly_steam_plants_sched402,1996,1395 ``` -------------------------------- ### Preview resource description via command line Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/metadata.rst Use the resource_description command to resolve and print metadata sections for a specific table without rendering the full Jinja template. ```bash $ resource_description -n core_ferc714__hourly_planning_area_demand Table found: core_ferc714__hourly_planning_area_demand Summary [timeseries[hourly]]: Hourly time series of electricity demand by planning area. Availability [True]: 2024 Layer [core]: Data has been cleaned and organized into well-modeled tables that serve as building blocks for downstream wide tables and analyses. Source [ferc714]: FERC Form 714 -- Annual Electric Balancing Authority Area and Planning Area Report (Part III, Schedule 2a) PK [True]: respondent_id_ferc714, datetime_utc Warnings [2]: custom - The datetime_utc timestamps have been cleaned due to inconsistent datetime reporting. See below for additional details. ferc_is_hard - FERC data is notoriously difficult to extract cleanly, and often contains free-form strings, non-labeled total rows and lack of IDs. See `Notable Irregularities `_ for details. Details [True]: This table includes data from the pre-2021 CSV raw source as well as the newer 2021 through present XBRL raw source. This table includes three respondent ID columns: one from the CSV raw source, one from the XBRL raw source and another that is PUDL-derived that links those two source ID's together. This table has filled in source IDs for all records so you can select the full timeseries for a given respondent from any of these three IDs. An important caveat to note is that there was some cleaning done to the datetime_utc timestamps. The Form 714 includes sparse documentation for respondents for how to interpret timestamps - the form asks respondents to provide 24 instances of hourly demand for each day. The form is labeled with hour 1-24. There is no indication if hour 1 begins at midnight. The XBRL data contained several formats of timestamps. Most records corresponding to hour 1 of the Form have a timestamp with hour 1 as T1. About two thirds of the records in the hour 24 location of the form have a timestamp with an hour reported as T24 while the remaining third report this as T00 of the next day. T24 is not a valid format for the hour of a datetime, so we convert these T24 hours into T00 of the next day. A smaller subset of the respondents reports the 24th hour as the last second of the day - we also convert these records to the T00 of the next day. ``` -------------------------------- ### Access gcloud storage help Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/nightly_data_builds.rst Use this command to view the CLI documentation for gcloud storage. ```bash gcloud storage --help ``` -------------------------------- ### Row Count Diff Example: New Data Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/data_validation_reference.rst Example diff showing the addition of a new year of data in the row counts CSV. ```diff diff --git a/dbt/seeds/etl_full_row_counts.csv b/dbt/seeds/etl_full_row_counts.csv index d9a5f0ec7..2b40f3ad7 100644 --- a/dbt/seeds/etl_full_row_counts.csv +++ b/dbt/seeds/etl_full_row_counts.csv @@ -3318,7 +3318,7 @@ out_ferc1__yearly_steam_plants_fuel_sched402,2020,1250 out_ferc1__yearly_steam_plants_fuel_sched402,2021,1152 out_ferc1__yearly_steam_plants_fuel_sched402,2022,1196 out_ferc1__yearly_steam_plants_fuel_sched402,2023,1210 +out_ferc1__yearly_steam_plants_fuel_sched402,2024,1221 out_ferc1__yearly_steam_plants_sched402,1994,1411 out_ferc1__yearly_steam_plants_sched402,1995,1448 out_ferc1__yearly_steam_plants_sched402,1996,1395 ``` -------------------------------- ### Initialize Analysis Environment Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/github-action-usage.ipynb Imports necessary libraries for data manipulation. ```python import pandas as pd import json ``` -------------------------------- ### Convert Years to Datetime Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/splink-ferc1-eia-match.ipynb Convert installation and construction year columns to datetime objects. ```python ferc_df["installation_year"] = pd.to_datetime(ferc_df["installation_year"], format="%Y") ferc_df["construction_year"] = pd.to_datetime(ferc_df["construction_year"], format="%Y") eia_df["installation_year"] = pd.to_datetime(eia_df["installation_year"], format="%Y") eia_df["construction_year"] = pd.to_datetime(eia_df["construction_year"], format="%Y") ``` -------------------------------- ### Initialize PUDL Engine Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/ferc1-eia-glue/training_data/create_FERC1-EIA_manual_mapping_spreadsheets.ipynb Sets up the database engine and PUDL output object for data retrieval. ```python pudl_engine = sa.create_engine(PudlPaths().pudl_db) pudl_out = pudl.output.pudltabl.PudlTabl(pudl_engine, freq='AS', fill_net_gen=True) ``` -------------------------------- ### Configure Notebook Environment Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/debug-eia-etl.ipynb Sets up autoreload, logging, and pandas display options for the debugging session. ```python %load_ext autoreload %autoreload 3 import logging import sys from pathlib import Path import pandas as pd import pudl pd.options.display.max_columns = None ``` ```python logger = logging.getLogger() logger.setLevel(logging.INFO) handler = logging.StreamHandler(stream=sys.stdout) formatter = logging.Formatter("%(message)s") handler.setFormatter(formatter) logger.handlers = [handler] ``` -------------------------------- ### Import required libraries Source: https://github.com/catalyst-cooperative/pudl/blob/main/notebooks/work-in-progress/epacems_regression_test.ipynb Initializes the environment with Polars and PUDL path utilities. ```python import polars as pl from polars.testing import assert_frame_equal from pudl.workspace.setup import PudlPaths ``` -------------------------------- ### Calculate Job Durations Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/github-action-usage.ipynb Computes the duration of each job by subtracting start time from completion time. ```python jobs_df["duration"] = jobs_df.completed_at - jobs_df.started_at ``` -------------------------------- ### Enable Git Pre-commit Hooks Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/dev_setup.rst Run this command to install and enable the pre-commit hook scripts within the PUDL pixi environment. ```console $ pixi run prek-install ``` -------------------------------- ### Initialize Name Cleaners Source: https://github.com/catalyst-cooperative/pudl/blob/main/devtools/splink-ferc1-eia-match.ipynb Set up the utility and plant name cleaning objects for string normalization. ```python plant_name_cleaner = eia_ferc1_model.plant_name_cleaner utility_name_cleaner = CompanyNameCleaner(legal_term_location=2) ``` -------------------------------- ### Sorted Response Data Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Example of the JSON structure returned when results are sorted by the period column. ```json response: { … [ { … period: "2021-11," }, { … period: "2021-10," }, { … period: "2021-09," }, … ] … } ``` -------------------------------- ### API Warning Response Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Example of a warning returned when a request exceeds the maximum row limit. ```json { warning: "parameter out of range" description: "The API can only return 5000 rows in JSON format. Please consider constraining your request with facet, start, or end, or using offset to paginate results." } ``` -------------------------------- ### Authenticate with Google Cloud SDK Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/dev/nightly_data_builds.rst Commands to authenticate and initialize the Google Cloud SDK for accessing internal build logs and data. ```bash gcloud auth login ``` ```bash gcloud init ``` ```bash gcloud auth application-default login ``` -------------------------------- ### API Error Response Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Example of a 400 error response returned when an invalid parameter is provided. ```json { error: "Invalid frequency 'millenially' provided. The only valid frequencies are 'monthly', 'quarterly', and 'annual'.", code: 400 } ``` -------------------------------- ### Initialize Logging Source: https://github.com/catalyst-cooperative/pudl/blob/main/notebooks/work-in-progress/eia861-transform.ipynb Configures the logging module to output information to standard output. ```python logger=logging.getLogger() logger.setLevel(logging.INFO) handler = logging.StreamHandler(stream=sys.stdout) formatter = logging.Formatter('%(message)s') handler.setFormatter(formatter) logger.handlers = [handler] ``` -------------------------------- ### GET /v2/{route} Source: https://github.com/catalyst-cooperative/pudl/blob/main/docs/data_sources/eiaapi/eiaapi_technical_documentation_2025-08-26.html Retrieve data from the EIA APIv2 using a hierarchical route structure. ```APIDOC ## GET /v2/{route} ### Description Access data series organized in a tree-like hierarchy using a RESTful route. ### Method GET ### Endpoint https://api.eia.gov/v2/{route} ### Parameters #### Query Parameters - **api_key** (string) - Required - The unique API key assigned to the user. ```