IPUMS Py

https://github.com/ipums/ipumspy
Admin
IPUMS Py is a collection of tools for working with data from IPUMS, enabling seamless access to and...
Tokens:40,326
Snippets:296
Trust Score:7.8
License:MPL-2.0
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Install ipumspy with pip

Source: https://github.com/ipums/ipumspy/blob/master/README.md

Install the ipumspy package. Requires Python 3.8 or later.

```bash
pip install ipumspy
```

--------------------------------

### Initialize IPUMS API client

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/getting_started.rst

Create an IpumsApiClient instance using an API key stored in the IPUMS_API_KEY environment variable. The example also imports necessary modules and sets up the download directory.

```python
import os
from pathlib import Path
from ipumspy import IpumsApiClient, MicrodataExtract, readers, ddi

# This assumes you have set up an environmental variable called 
# "IPUMS_API_KEY" to store your IPUMS API key
IPUMS_API_KEY = os.environ.get("IPUMS_API_KEY")

ipums = IpumsApiClient(IPUMS_API_KEY)
```

--------------------------------

### YAML extract definition file

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Example YAML file containing an extract definition for the USA collection with specified samples and variables.

```yaml
description: Sample extract
collection: usa
samples:
  - us2018a
  - us2019a
variables:
  - AGE
  - SEX
  - RACE
  - STATEFIP
  - MARST
```

--------------------------------

### Submit, wait, and download extract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/getting_started.rst

Submit the extract to IPUMS servers, wait for processing to complete, and download the data to a specified directory. The example prints the extract ID after submission.

```python
# Submit the extract request
ipums.submit_extract(extract)
print(f"Extract submitted with id {extract.extract_id}")
#> Extract submitted with id 1

# Wait for the extract to finish
ipums.wait_for_extract(extract)

# Download the extract
DOWNLOAD_DIR = Path(<your_download_dir>)
ipums.download_extract(extract, download_dir=DOWNLOAD_DIR)
```

--------------------------------

### Install ipumspy with conda

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/getting_started.rst

Install the ipumspy package using conda from the conda-forge channel. Requires Python 3.10 or higher.

```bash
conda install -c conda-forge ipumspy
```

--------------------------------

### Create a CPS MicrodataExtract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Initializes a MicrodataExtract for IPUMS CPS data with specified samples, variables, and a description. This is the base extract used in the examples that follow.

```python
extract = MicrodataExtract(
    collection="cps",
    samples=["cps2022_03s"],
    variables=["AGE", "SEX", "RACE"],
    description="A CPS extract example"
)
```

--------------------------------

### Include shapefiles in an NHGIS extract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

This example includes shapefiles in an NHGIS extract by specifying their names. Shapefiles have no additional selection parameters, so you only need to provide the shapefile names.

```python
AggregateDataExtract(
   collection="nhgis",
   shapefiles=["us_county_2021_tl2021", "us_county_2020_tl2020"]
)
```

--------------------------------

### Read microdata and get variable info for SEX

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/reading_data.rst

Reads a DDI codebook and microdata file, then retrieves the VariableDescription for the SEX variable and prints its codes and description.

```python
from ipumspy import readers
import pandas as pd

# read ddi and data
ddi_codebook = readers.read_ipums_ddi(path/to/ddi/xml/file)
ipums_df = readers.read_microdata(ddi_codebook, path/to/data/file)

# get VariableDescription for SEX
sex_info = ddi_codebook.get_variable_info("SEX")

# see codes and labels for SEX
print(sex_info.codes)
#> {'Male': 1, 'Female': 2}

# see variable description for SEX
print(sex_info.description)
#> SEX reports whether the person was male or female.
```

--------------------------------

### Create MicrodataExtract with sample members

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Use this snippet to include household members of ATUS respondents and non-respondents in an IPUMS ATUS extract. The sample_members keyword argument controls inclusion; ATUS respondents are included by default. This example also includes time use variables.

```python
atus_extract = MicrodataExtract(
    collection="atus",
    samples=["at2016"],
    variables=["AGE", "SEX"],
    time_use_variables=[
        TimeUseVariable(name="BLS_PCARE"), 
        TimeUseVariable(name="MY_CUSTOM_TUV", owner="newipumsuser@gmail.com")
    ],
    sample_members={
        "include_non_respondents": True,
        "include_household_members": True
    },
    description="Sample members example"
)
```

--------------------------------

### Create a USA MicrodataExtract for Case Selection

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Initializes a MicrodataExtract for IPUMS USA data with case selection enabled at the household level. This extract is used in the detailed codes example.

```python
extract = MicrodataExtract(
    collection="usa",
    samples=["us2021a"],
    variables=["AGE", "SEX", "RACE"],
    description="Case selection example",
    case_select_who = "households"
)
```

--------------------------------

### Hierarchical Data Structure Example

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Requests a hierarchical data structure for a USA extract by setting data_structure to {"hierarchical": {}}. The default data_structure is rectangular on person, so this overrides that default.

```python
extract = MicrodataExtract(
    collection="usa",
    samples=["us2012b"],
    variables=["AGE", "SEX"],
    description="An IPUMS extract example",
    data_structure={"hierarchical": {}},
)
```

--------------------------------

### Create MicrodataExtract with time use variables

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Use this snippet to create an IPUMS ATUS extract with user-defined time use variables. The owner field must contain the email associated with your IPUMS account. The example includes standard variables AGE and SEX along with two time use variables.

```python
atus_extract = MicrodataExtract(
    collection="atus",
    samples=["at2016"],
    variables=["AGE", "SEX"],
    time_use_variables=[
        TimeUseVariable(name="BLS_PCARE"), 
        TimeUseVariable(name="MY_CUSTOM_TUV", owner="newipumsuser@gmail.com")
    ],
    description="User-defined time use variable example"
)
```

--------------------------------

### Request multiple datasets and shapefiles in one extract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Creates an AggregateDataExtract for NHGIS that combines tabular datasets with shapefiles in a single request. The example requests state-level population totals from the 2000 and 2010 decennial censuses along with corresponding state boundary shapefiles.

```python
# Total state-level population from 2000 and 2010 decennial census
   extract = AggregateDataExtract(
      collection="nhgis",
      description="An NHGIS example extract",
      datasets=[
         NhgisDataset(name="2000_SF1a", data_tables=["NP001A"], geog_levels=["state"]),
         NhgisDataset(name="2010_SF1a", data_tables=["P1"], geog_levels=["state"])
      ],
      shapefiles=["us_state_2000_tl2010", "us_state_2010_tl2010"]
   )
```

--------------------------------

### Create a MicrodataExtract definition

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/getting_started.rst

Define an extract request for IPUMS USA data, specifying collection, description, samples, and variables. This example requests 2012 Puerto Rico Community Survey data for age and sex.

```python
# Create an extract definition
extract = MicrodataExtract(
    collection="usa",
    description="Sample USA extract",
    samples=["us2012b"],
    variables=["AGE", "SEX"],
)
```

--------------------------------

### Select Cases with Detailed Codes

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Selects cases using detailed codes for the RACE variable, setting general=False to use detailed codes instead of general ones. This example selects respondents who identify as White and Asian.

```python
extract.select_cases("RACE", 
                         ["810", "811", "812", "813", "814", "815", "816", "818"], 
                         general=False)
```

--------------------------------

### Get NHGIS dataset metadata with NhgisDatasetMetadata

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Fetches metadata for a specific NHGIS dataset using the IPUMS Metadata API. Requires an API key stored in the IPUMS_API_KEY environment variable.

```python
from ipumspy import IpumsApiClient, NhgisDatasetMetadata

ipums = IpumsApiClient(os.environ.get("IPUMS_API_KEY"))

ds = ipums.get_metadata(NhgisDatasetMetadata("2000_SF1a"))
```

--------------------------------

### Read hierarchical IPUMS extract with read_hierarchical_microdata

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/reading_data.rst

Reads a hierarchical extract and returns a dictionary of data frames keyed by record type. The path in the example contains a typo (hierarhcical) as shown in the source.

```python
extract_dict = readers.read_hierarchical_microdata(ddi, 
                                                   path/to/hierarhcical_file.dat.gz)
```

--------------------------------

### Add time series tables to an NHGIS extract with TimeSeriesTable

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

These examples show how to add time series tables to an NHGIS extract using the TimeSeriesTable class. The first creates a basic request with default all years; the second selects specific years with the years argument; the third sets the time series layout with tst_layout. Time series tables do not require data table selection, but you must specify geog_levels.

```python
from ipumspy import AggregateDataExtract, TimeSeriesTable

extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS example extract: time series tables",
   time_series_tables=[TimeSeriesTable("CW3", geog_levels=["county", "state"])],
)
```

```python
extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS example extract: time series tables",
   time_series_tables=[
      TimeSeriesTable("CW3", geog_levels=["county", "state"], years=[1990, 2000])
   ],
)
```

```python
extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS example extract: time series tables",
   time_series_tables=[
      TimeSeriesTable("CW3", geog_levels=["county", "state"], years=[1990, 2000])
   ],
   tst_layout="time_by_row_layout",
)
```

--------------------------------

### Limit NHGIS extract to geographic extents with geographic_extents

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

This example requests ACS 5-year sex-by-age counts at the census block group level, but only includes block groups within Alabama and Arkansas (FIPS codes with trailing 0). The geographic_extents argument is only available for NHGIS extracts and applies to all datasets and time series tables in the extract.

```python
extract = AggregateDataExtract(
   collection="nhgis",
   description="Extent selection example",
   datasets=[
      NhgisDataset(name="2018_2022_ACS5a", data_tables=["B01001"], geog_levels=["blck_grp"]),
      NhgisDataset(name="2017_2021_ACS5a", data_tables=["B01001"], geog_levels=["blck_grp"])
   ],
   geographic_extents=["010", "050"]
)
```

--------------------------------

### ipums --help

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Displays available commands for the ipums CLI.

```bash
ipums --help
```

--------------------------------

### ipums submit --help

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Shows extra options for the submit command, such as -k for API key and -o for download directory.

```bash
ipums submit --help
```

--------------------------------

### Submit metadata request and display description

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Submits the metadata request with get_metadata and prints the description of the returned object.

```python
ipums.get_metadata(tst)

tst.description
#> 'Total Population'
```

--------------------------------

### Download extract to a directory

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Downloads the extract to the specified download directory.

```python
DOWNLOAD_DIR = Path("<your_download_dir>")
ipums.download_extract(extract, download_dir=DOWNLOAD_DIR)
```

--------------------------------

### Step-by-step extract submission, check, and download

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Commands to submit an extract, check its status, and download it. The check command shows status messages; wait until status is 'completed' before downloading.

```bash
ipums submit -k <IPUMS_API_KEY> ipums.yaml
# Your extract for collection usa has been successfully submitted with number 10

ipums check -k <IPUMS_API_KEY> 10
# Extract 10 in collection usa has status started

# "started" means that your extract has been queued
# You should wait until the status is "completed"
ipums check -k <IPUMS_API_KEY> 10
# Extract 10 in collection usa has status completed

ipums download -k <IPUMS_API_KEY> 10
```

--------------------------------

### Wait for extract to complete

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Blocks until the extract is ready for download.

```python
ipums.wait_for_extract(extract)
```

--------------------------------

### Request time use variables in ATUS extract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Creates an ATUS extract that includes standard variables AGE and SEX plus the system time use variable BLS_PCARE. Time use variables must be passed via the time_use_variables argument, not the variables list.

```python
atus_extract = MicrodataExtract(
    collection="atus",
    samples=["at2016"],
    variables=["AGE", "SEX"],
    time_use_variables=["BLS_PCARE"],
    description="An time use variable example"
)
```

--------------------------------

### Wait for and download extract using collection and extract_id

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Waits for the extract to complete and then downloads it using only the collection name and extract ID.

```python
ipums.wait_for_extract(extract=1, collection="usa")
ipums.download_extract(extract=1, collection="usa")
```

--------------------------------

### Load downloaded microdata with readers

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/getting_started.rst

After downloading an extract, locate the DDI XML file in the download directory, parse it, and load the microdata into a pandas DataFrame using the readers module.

```python
# Get the DDI
ddi_file = list(DOWNLOAD_DIR.glob("*.xml"))[0]
ddi = readers.read_ipums_ddi(ddi_file)

# Get the data
ipums_df = readers.read_microdata(ddi, DOWNLOAD_DIR / ddi.file_description.filename)
```

--------------------------------

### ipums submit-and-download with ipums_with_race.yaml

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Submits and downloads the extract defined in ipums_with_race.yaml.

```bash
ipums submit-and-download -k <IPUMS_API_KEY> ipums_with_race.yaml
```

--------------------------------

### Download NHGIS supplemental data with IpumsApiClient.get

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Downloads a supplemental NHGIS crosswalk file directly via the IPUMS API. Requires an API key stored in the IPUMS_API_KEY environment variable and a known file URL. The file is streamed to the specified download path.

```python
ipums = IpumsApiClient(os.environ.get("IPUMS_API_KEY"))

file_name = "nhgis_blk2010_blk2020_10.zip"
url = f"{ipums.base_url}/supplemental-data/nhgis/crosswalks/nhgis_blk2010_blk2020_state/{file_name}"

download_path = "<your-download-path-here>"

with ipums.get(url, stream=True) as response:
   with open(download_path, "wb") as outfile:
      for chunk in response.iter_content(chunk_size=8192):
            outfile.write(chunk)
```

--------------------------------

### ipums submit-and-download with ipums_multiple.yaml

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Submits and downloads multiple extracts defined in ipums_multiple.yaml.

```bash
ipums submit-and-download -k <IPUMS_API_KEY> ipums_multiple.yaml
```

--------------------------------

### Load YAML extract definition

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Loads a YAML extract definition file, parses it into a dictionary, and converts it to a MicrodataExtract object using extract_from_dict.

```python
import yaml
from ipumspy import extract_from_dict

with open("ipums.yaml") as infile:
    extract = extract_from_dict(yaml.safe_load(infile))
```

--------------------------------

### ipums submit-and-download with ipums.yaml

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Submits, waits for, and downloads an extract defined in ipums.yaml using the IPUMS_API_KEY.

```bash
ipums submit-and-download -k <IPUMS_API_KEY> ipums.yaml
```

--------------------------------

### Load parquet file with ipumspy

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Python code to read a DDI and microdata from a parquet file using ipumspy.

```python
import ipumspy

ddi = ipumspy.read_ipums_ddi(ddi_file)
ipums_df = ipumspy.read_microdata(ddi, "usa_00006.parquet")
```

--------------------------------

### Load JSON extract definition and submit

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Loads a JSON extract definition file, creates an extract object, and submits it to the IPUMS API. Requires an API key and the collaborator must be registered with the data collection.

```python
import os
from ipumspy import IpumsApiClient, define_extract_from_json

IPUMS_API_KEY = os.environ.get("IPUMS_API_KEY")
ipums = IpumsApiClient(IPUMS_API_KEY)

extract = define_extract_from_json("my_extract.json")
ipums.submit_extract(extract)
```

--------------------------------

### Request multi-year NHGIS dataset with years

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Requests data from a multi-year NHGIS dataset (1988_1997_CBPa) by specifying the years parameter. Use years=["*"] to select all years in a dataset.

```python
extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS example extract",
   datasets=[
      NhgisDataset(
         name="1988_1997_CBPa",
         data_tables=["NT004"],
         geog_levels=["county"],
         years=[1988, 1989, 1990],
     )
   ],
)
```

--------------------------------

### Create an IHGIS extract with AggregateDataExtract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Creates an IHGIS extract request. Only the description and datasets parameters are accepted; other AggregateDataExtract arguments are omitted if included.

```python
AggregateDataExtract(
   collection="ihgis",
   description="An IHGIS example extract",
   datasets=[
      IhgisDataset(
         "KZ2009pop", 
         data_tables=["KZ2009pop.AAA"], 
         tabulation_geographies=["KZ2009pop.g0"]
      )
   ]
)
```

--------------------------------

### Request detailed metadata for NHGIS time series table A00

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Creates a TimeSeriesTableMetadata object for NHGIS table 'A00' to request detailed metadata.

```python
tst = TimeSeriesTableMetadata("nhgis", "A00")
```

--------------------------------

### Read compressed CSV from zip archive

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/reading_data.rst

Reads the first data file from the zip archive using pandas, demonstrating how to load compressed CSV data from an aggregate extract.

```python
# Read first data file in the extract
with ZipFile(fname) as z:
    with z.open(names[0]) as f:
        data = pd.read_csv(f)
```

--------------------------------

### Add data quality flags for specific variables

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Shows how to add data quality flags for individual variables using the add_data_quality_flags() method. Accepts a single variable name or a list of variable names.

```python
# add the data quality flag for AGE to the extract
extract.add_data_quality_flags("AGE")

# note that this method will also accept a list!
extract.add_data_quality_flags(["AGE", "SEX"])
```

--------------------------------

### Attach Characteristics for Spouse

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Uses the attach_characteristics() method to create a new variable SEX_SP that contains the spouse's AGE value, or 0 if no spouse. Valid household members include 'spouse', 'mother', 'father', and 'head'.

```python
extract.attach_characteristics("SEX", ["spouse"])
```

--------------------------------

### ipums convert to parquet

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

Converts a downloaded extract (usa_00006.dat) and its DDI (usa_00006.xml) to a parquet file (usa_00006.parquet).

```bash
ipums convert usa_00006.xml usa_00006.dat usa_00006.parquet
```

--------------------------------

### Retrieve previous extracts with get_previous_extracts()

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Retrieves the most recent extracts for a given collection. By default returns 10 extracts, but the limit argument can adjust the count.

```python
from ipumspy import IpumsApiClient

ipums = IpumsApiClient("YOUR_API_KEY")

# get my 10 most-recent USA extracts
recent_extracts = ipums.get_previous_extracts("usa")

# get my 20 most-recent CPS extracts
more_recent_extracts = ipums.get_previous_extracts("cps", limit=20)
```

--------------------------------

### Basic MicrodataExtract for IPUMS USA

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Constructs a MicrodataExtract for the IPUMS USA collection with the us2012b sample and AGE/SEX variables. The description is optional but recommended for later identification. After instantiation, the object can be submitted to the API for processing.

```python
extract = MicrodataExtract(
    collection="usa",
    samples=["us2012b"],
    variables=["AGE", "SEX"],
    description="An IPUMS extract example"
)
```

--------------------------------

### Generate dataset specs for multiple ACS years with list comprehension

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Builds a list of NhgisDataset specifications for multiple ACS years using a list comprehension. Useful when data table codes are consistent across datasets, as with ACS tables like B01001.

```python
acs1_names = ["2017_ACS1", "2018_ACS1", "2019_ACS1"]
   acs1_specs = [
      NhgisDataset(name, data_tables=["B01001"], geog_levels=["state"]) for name in acs1_names
   ]
```

--------------------------------

### Print IPUMS citation from DDI

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/index.rst

Use this snippet to display the recommended citation for an IPUMS extract. The citation is stored in the DDI's ipums_citation attribute.

```python
print(ddi.ipums_citation)
```

--------------------------------

### Create NHGIS aggregate extract with AggregateDataExtract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Creates an NHGIS aggregate extract for state-level population from 2017-2019 ACS 1-year estimates. The collection is set to "nhgis" and the datasets are provided via acs1_specs.

```python
extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS example extract",
   datasets=acs1_specs,
)
```

--------------------------------

### Extract with data_format Argument

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Specifies a desired file format (e.g., CSV) when creating an extract object by using the data_format argument. This is an alternative to the default format.

```python
extract = MicrodataExtract(
    collection="usa",
    samples=["us2012b"],
    variables=["AGE", "SEX"],
    description="An IPUMS extract example",
    data_format="csv",
)
```

--------------------------------

### Create an AggregateDataExtract for NHGIS

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Constructs an AggregateDataExtract for the IPUMS NHGIS collection, requesting county-level data from tables NP1 and NP2 of the 1990 STF 1 decennial census file. The extract must include a collection ID and at least one data source; a description is recommended for future identification.

```python
from ipumspy import AggregateDataExtract, NhgisDataset

extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS extract example",
   datasets=[
      NhgisDataset(name="1990_STF1", data_tables=["NP1", "NP2"], geog_levels=["county"])
   ]
)
```

--------------------------------

### Filter CPS extract history for STATEFIP variable

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Use get_extract_history() with page_size=100 to iterate through CPS extract history pages, parse each extract definition, and collect extracts containing the STATEFIP variable. The generator returns pages of up to 500 extract definitions by default, but page_size can be lowered.

```python
extracts_with_state = []

# Get pages with 100 CPS extracts per page
for page in ipums.get_extract_history("cps", page_size=100):
    for ext in page["data"]:
        extract_obj = extract_from_dict(ext["extractDefinition"])
        if "STATEFIP" in [var.name for var in extract_obj.variables]:
            extracts_with_state.append(extract_obj)
```

--------------------------------

### Add data quality flags to all variables

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Creates a CPS MicrodataExtract with data_quality_flags=True to add all available data quality flags for the specified variables. Flags are only added if they exist in the extract's sample(s).

```python
extract = MicrodataExtract(
    collection="cps",
    samples=["cps2022_03s"],
    variables=["AGE", "SEX", "RACE"],
    data_quality_flags=True
)
```

--------------------------------

### Check extract expiration status with extract_status()

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Checks the status of an expired extract. Even if the extract data and syntax files have been purged, the status remains 'completed'.

```python
# Extract number 1 has expired, but status listed as completed
extract_status = ipums.extract_status(extract=1, collection="usa")

print(extract_status)
#> completed
```

--------------------------------

### Specify NHGIS dataset and data tables

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Specifies an NHGIS extract using the NhgisDataset class to request state-level data from tables NP001A and NP031A of the 2000_SF1a dataset. This is the standard way to define datasets and data tables for NHGIS extracts.

```python
extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS example extract",
   datasets=[
      NhgisDataset(name="2000_SF1a", data_tables=["NP001A", "NP031A"], geog_levels=["state"])
   ],
)
```

--------------------------------

### ipums_multiple.yaml multiple extracts specification

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

YAML configuration for submitting multiple extracts simultaneously using an 'extracts' key with a list of extract specifications.

```yaml
extracts:
- description: Simple IPUMS extract
  collection: usa
  samples:
    - us2012b
  variables:
    - AGE
    - SEX
- description: Another extract
  collection: usa
  samples:
    - us2012b
  variables:
    - AGE
    - SEX
    - RACE
```

--------------------------------

### Attach Characteristics for Parents

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Adds variables for the RACE value of both the mother and father using the attach_characteristics() method.

```python
extract.attach_characteristics("RACE", ["mother", "father"])
```

--------------------------------

### List files in aggregate data zip archive

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/reading_data.rst

Uses Python's zipfile module to list the names of files contained in an IPUMS aggregate data extract zip file.

```python
from zipfile import ZipFile
import pandas as pd

fname = "<path/to/zip/file>"

# a list of individual file names from inside the .zip file
names = ZipFile(fname).namelist()
#> ['nhgis0025_csv/nhgis0025_ds120_1990_county_codebook.txt', 'nhgis0025_csv/nhgis0025_ds120_1990_county.csv']
```

--------------------------------

### Read IPUMS data in chunks with read_microdata_chunked

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/reading_data.rst

Reads the microdata in chunks of 1000 rows and filters for Minnesota (STATEFIP == 27) using pandas concat. Useful for large files that do not fit in memory.

```python
iter_microdata = read_microdata_chunked(ddi, chunksize=1000)
df = pd.concat([df[df["STATEFIP"] == 27] for df in iter_microdata])
```

--------------------------------

### ipums.yaml extract specification

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

YAML configuration for a simple IPUMS extract, specifying collection, samples, and variables.

```yaml
description: Simple IPUMS extract
collection: usa
samples:
  - us2012b
variables:
  - AGE
  - SEX
```

--------------------------------

### Find NHGIS data tables with 'Urban Population' in description

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Iterates through metadata catalog pages to collect data tables whose description contains 'Urban Population'. Use with get_metadata_catalog for NHGIS data_tables.

```python
urb_dts = []

# Identify all data tables referring to "Urban Population"
for page in ipums.get_metadata_catalog("nhgis", metadata_type="data_tables"):
    for dt in page["data"]:
        if "Urban Population" in dt["description"]:
            urb_dts.append(dt)
```

--------------------------------

### Save extract definition as JSON

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Saves an extract object to a JSON file for sharing with collaborators.

```python
save_extract_as_json(extract, filename="my_extract.json")
```

--------------------------------

### Check extract status using collection and extract_id

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Checks the status of an extract using only the collection name and extract ID.

```python
# Check the extract status
extract_status = ipums.extract_status(extract=1, collection="usa")
print(f"extract is {extract_status}")
#> extract is started
```

--------------------------------

### Select Cases with General Codes

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Selects only female records (code "2") from the CPS extract using the select_cases() method. Note that select_cases can only be used with categorical variables already present in the extract.

```python
extract.select_cases("SEX", ["2"])
```

--------------------------------

### Read IPUMS microdata extract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/getting_started.rst

Parse a DDI XML codebook and load the corresponding microdata file into a pandas DataFrame. Supports both fixed-width and CSV files.

```python
from ipumspy import readers, ddi

ddi_codebook = readers.read_ipums_ddi(ddi/xml/file path/)
ipums_df = readers.read_microdata(ddi_codebook, data/file/path)
```

--------------------------------

### Adjust monetary values to 2010 dollars

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Creates a CPS extract and adjusts the HOURWAGE variable to 2010 dollars using adjust_monetary_values(). The original variable is retained and a new variable named HOURWAGE_cpiu_2010 is added. Not available for IPUMS CPS ASEC variables.

```python
extract = MicrodataExtract(
    collection="cps",
    samples=["cps2022_03b"],
    variables=["AGE", "SEX", "HOURWAGE"],
)

extract.adjust_monetary_values("HOURWAGE")
```

--------------------------------

### Submit extract request

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Submits an extract object to the IPUMS servers for processing.

```python
ipums.submit_extract(extract)
```

--------------------------------

### Create MicrodataExtract for IPUMS USA variables

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Defines a simple IPUMS USA microdata extract request for the AGE, SEX, RACE, STATEFIP, and MARST variables from the 2018 and 2019 American Community Survey (ACS). The collection is set to "usa" and the samples are "us2018a" and "us2019a".

```python
extract = MicrodataExtract(
    collection="usa",
    description="Sample USA extract",
    samples=["us2018a", "us2019a"],
    variables=["AGE", "SEX", "RACE", "STATEFIP", "MARST"],
)
```

--------------------------------

### Filter data using value label for Female

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/reading_data.rst

Filters the microdata data frame to retain only female respondents by comparing the SEX column to the numeric code for the 'Female' label.

```python
# retrieve the VaribleDescription for the variable SEX
sex_info = ddi_codebook.get_variable_info("SEX")

# Filter to records where SEX is Female
women = ipums_df[ipums_df["SEX"] == sex_info.codes["Female"]]
```

--------------------------------

### Filter data using value label and numeric condition

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/reading_data.rst

Filters the data frame to retain only women over the age of 16 by combining a label-based condition for SEX with a numeric condition on AGE.

```python
adult_women = ipums_df[(ipums_df["SEX"] == sex_info.codes["Female"]) &
                           (ipums_df["AGE"] > 16)]
```

--------------------------------

### Request NHGIS extract with breakdown_values

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_aggregate/index.rst

Optionally requests specific breakdown values for a dataset using the breakdown_values keyword argument. By default, the first available breakdown (typically the total count) is selected; when retrieving a previously submitted extract, a breakdown value code may appear even if not explicitly requested.

```python
extract = AggregateDataExtract(
   collection="nhgis",
   description="An NHGIS example extract",
   datasets=[
      NhgisDataset(
            name="2000_SF1a",
            data_tables=["NP001A", "NP031A"],
            geog_levels=["state"],
            breakdown_values=["bs21.ge01", "bs21.ge43"],  # Urban + Rural breakdowns
      )
   ],
)
```

--------------------------------

### Print extract ID after submission

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Prints the extract ID assigned to a successfully submitted extract.

```python
print(extract.extract_id)
#> 1
```

--------------------------------

### ipums_with_race.yaml extract specification

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/cli.rst

YAML configuration for an extract that includes the RACE variable.

```yaml
description: Another extract
collection: usa
samples:
  - us2012b
variables:
  - AGE
  - SEX
  - RACE
```

--------------------------------

### Re-create and re-submit an expired extract

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Re-creates a MicrodataExtract object from an expired extract definition and submits it to generate new data and syntax files. The new extract receives a new unique ID.

```python
# Create a MicrodataExtract object from the expired extract definition
renewed_extract = ipums.get_extract_by_id(collection="usa", extract_id=1)

# Submit the renewed extract to re-generate the data and syntax files
resubmitted_extract = ipums.submit_extract(renewed_extract)

print(resubmitted_extract.extract_id)
#> 2
```

--------------------------------

### Define extract with Variable objects

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Defines a CPS extract using Variable objects to set per-variable features: attached characteristics, case selections, data quality flags, and monetary value adjustment. The description field is also set.

```python
fancy_extract = MicrodataExtract(
    collection="cps",
    samples=["cps2022_03s"],
    variables=[
        Variable(name="AGE",
                 attached_characteristics=["spouse"]),
        Variable(name="SEX",
                 case_selections={"general": ["2"]}),
        Variable(name="RACE",
                 data_quality_flags=True),
        Variable(name="HOURWAGE",
                 adjust_monetary_values=True),
     ],
     description="A fancy CPS extract",
)
```

--------------------------------

### Confirm extract expiration with extract_is_expired()

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Confirms whether an extract has expired. Returns True if the extract's data and syntax files have been purged from the IPUMS cache.

```python
is_expired = ipums.extract_is_expired(extract=1, collection="usa")

print(is_expired)
#> True
```

--------------------------------

### Rectangular Extract on Round for IPUMS MEPS

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/ipums_api_micro/index.rst

Creates a MEPS extract with a rectangular data structure on round records by setting data_structure to {"rectangular": {"on": "R"}}. This is used when the collection supports rectangular files on non-person record types.

```python
extract = MicrodataExtract(
    collection="meps",
    samples=["mp2016"],
    variables=["AGE", "SEX", "PREGNTRD"],
    description="An IPUMS extract example",
    data_structure={"rectangular": {"on": "R"}},
)
```

--------------------------------

### Check extract status

Source: https://github.com/ipums/ipumspy/blob/master/docs/source/ipums_api/index.rst

Checks the current status of an extract request using the extract object.

```python
print(ipums.extract_status(extract))
#> started
```
Tokens:
Raw