### Fetching Subset Data for PUMS Dataset (Git)

Source: https://github.com/arrowrbook/book/blob/main/writers notes/README.md

This snippet provides Git commands to fetch and extract a small subset of the PUMS dataset (`PUMS_smol`) from a specific Git branch (`CI_data`) into the local repository. This subset is used for faster development and iteration cycles. It ensures the `PUMS_smol` directory is placed in the repo without overlaying existing files and then unstages it.

```Shell
git fetch origin CI_data:CI_data
git checkout --no-overlay CI_data -- PUMS_smol
git restore --staged PUMS_smol
```

--------------------------------

### Syncing Full PUMS Dataset from S3 (AWS CLI)

Source: https://github.com/arrowrbook/book/blob/main/writers notes/README.md

This command uses the AWS S3 CLI to synchronize the full PUMS dataset from a specified S3 bucket (`s3://scaling-arrow-pums`) to a local directory. This is intended for storing the complete, large dataset (over 75GB) required for full book rendering and publication. The `{some path where you can hold this data}` placeholder should be replaced with the desired local storage path.

```Shell
aws s3 sync s3://scaling-arrow-pums {some path where you can hold this data}
```

--------------------------------

### Downloading and Processing PUMS Data in R

Source: https://github.com/arrowrbook/book/blob/main/pums_dataset/README.md

This R snippet defines the download path for PUMS data and sets the number of cores for parallel processing. It then sequentially executes scripts to download raw PUMS data, unzip it, recode variables, and repartition the dataset for optimized use, typically with Arrow.

```R
dl_path <- "~/PUMS"
options(mc.cores = ...)
source("pums_dataset/download_raw_pums.R")
source("pums_dataset/unzip_raw_pums.R")
source("pums_dataset/recode_pums.R")
source("pums_dataset/repartition_pums.R")
```

--------------------------------

### Uploading Processed PUMS Data to AWS S3 using Bash

Source: https://github.com/arrowrbook/book/blob/main/pums_dataset/README.md

This Bash command uses the AWS S3 CLI to synchronize the current directory (assumed to be the root of the cleaned dataset) with an S3 bucket. The --dryrun flag allows for a preview of changes, --delete removes files in S3 that are not locally present, and --exclude prevents specific files like .DS_Store from being uploaded.

```Bash
aws s3 sync --dryrun --delete --exclude "*.DS_Store" . s3://scaling-arrow-pums
```

--------------------------------

### Creating a Small PUMS Subset in Bash

Source: https://github.com/arrowrbook/book/blob/main/pums_dataset/README.md

This Bash command sources an R script responsible for generating a smaller, more manageable subset of the PUMS data. This 'smol' subset is typically used for local development or CI/CD pipelines and is often kept under version control in a dedicated branch.

```Bash
source("pums_dataset/make_smol_pums.R")
```

--------------------------------

### Generating PUMS Metadata in R

Source: https://github.com/arrowrbook/book/blob/main/pums_dataset/README.md

This R snippet sets the CENSUS_KEY environment variable, which is required for accessing Census data, and then sources a script to download or regenerate the PUMS metadata. This step is crucial if the existing metadata file is incorrect or if new data years are added.

```R
Sys.setenv(CENSUS_KEY="...")
source("pums_dataset/download_pums_metadata.R")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.