OpenPoliceData

https://openpolicedata.readthedocs.io/en/stable/
Admin
OpenPoliceData (OPD) aims to make it easier to access and analyze police data by providing the most...
Tokens:37,688
Snippets:474
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install OpenPoliceData

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Install the library using pip. Use %pip when installing within a Jupyter Notebook environment.

```bash
pip install openpolicedata
```

```bash
%pip install openpolicedata
```

--------------------------------

### Install OpenPoliceData via pip

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/installation.rst.txt

Standard installation command to enable most datasets.

```bash
pip install openpolicedata
```

--------------------------------

### Install OpenPoliceData with optional dependencies

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/installation.rst.txt

Installs additional packages like rapidfuzz and msoffcrypto-tool required for specific datasets.

```bash
pip install "openpolicedata[optional]"
```

--------------------------------

### Arrest Analysis Output Example

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Example output showing demographic counts and percentages for total cases versus repeat offenders.

```text
Counts for all cases (Total: 670):
H    304
B    200
W    147
U      9
A      7
I      3
Name: race_eth, dtype: int64


Counts for individuals arrests multiple times (Total: 68):
H    35
B    23
W     9
I     1
Name: race_eth, dtype: int64


Percentages for all cases (Total: 670):
H    45.4
B    29.9
W    21.9
U     1.3
A     1.0
I     0.4
Name: race_eth, dtype: float64


Percentages for individuals arrests multiple times (Total: 68):
H    51.5
B    33.8
W    13.2
I     1.5
Name: race_eth, dtype: float64
```

--------------------------------

### Install GeoPandas via conda

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/installation.rst.txt

Enables loading location-based data into GeoDataFrames for geospatial analysis.

```bash
conda install geopandas
```

--------------------------------

### Load multi-year dataset

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Load a dataset spanning multiple years by providing a list of start and end years to the date parameter.

```python
multiyear_tbl = src.load("FIELD CONTACTS", date=[2021, 2024])
multiyear_tbl.table['Year'].value_counts()
```

--------------------------------

### Get Available Years

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

List all years for which data is available for a given table type.

```python
src.get_years(table_type="FIELD CONTACTS")
```

--------------------------------

### Initialize Source and View Datasets

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/export_dataset_to_csv_and_other_file_types.html

Create a source object for a specific agency and state to list available datasets.

```python
# To access the data, create a source using a Source Name (usually a police department name). There is an optional state input to clarify ambiguities.
# We will use the above cell's information for Maryland to choose the agency "Montgomery County" which we select for the source_name

src = opd.Source(source_name="Montgomery County", state="Maryland")
src.datasets.head()
```

--------------------------------

### Initialize data source

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Creates a source object using the information retrieved from the query.

```python
src = opd.Source(state="Maryland", agency="Montgomery County")
```

--------------------------------

### Creating a Data Source and Listing Datasets

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Initializes a source object for Fairfax County and displays the available datasets for further analysis.

```python
# Create a source. Sources know what date is availabe in the package, where to find it, and how to load it.

# This source is for Fairfax County's open data
# The name to the left of the equals sign is the variable that is generated. You use this name when you want to use it below.
src = opd.Source("Fairfax County")

# Sources have a datasets property that is a table of available datasets from the source
# Print out the available datasets
# If you don't output to a variable, it will print out below. If we put x = src.datasets here, it would set the variable x to be
# src.datasets instead of printing it out

# In a Jupyter notebook, if you don't output something to a variable (i.e. x=src.datasets would output src.datasets to x), it will automatically print it out
# (NOTE: this can only be done once per cell. Only the last item that isn't outputted to the variable will be printed. To print multiple things, use the print function)
src.datasets
```

--------------------------------

### Get Record Count

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Retrieve the number of records for a specific table type and year.

```python
src.get_count("FIELD CONTACTS", 2022)
```

--------------------------------

### Initialize OPD and Helper Function

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Imports the OpenPoliceData library and defines a utility function for wrapping long text output.

```python
import openpolicedata as opd
import textwrap
# Convenience method for printing long text
def wrap_print(text, newline=True):
    text = "\n".join(textwrap.wrap(text, width=110))
    if newline:
        text = "\n"+text
    print(text)
```

--------------------------------

### Initialize a Data Source

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/index.ipynb.txt

Create a Source object for a specific location to view available datasets. Specifying the state is required when multiple locations share the same name.

```python
src = opd.Source(state="South Carolina", source_name="Columbia")
src.datasets
```

--------------------------------

### Requesting a Date Range for Multi-Year Datasets

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/date_filtering.ipynb.txt

Use a list containing start and stop dates in YYYY-MM-DD format as the date input to filter data.

```python
table = asheville.load('USE OF FORCE', ['2020-09-13', '2020-10-10'])
print(f"Earliest date in table is {table.table['date_occurred'].min()}")
print(f"Latest date in table is {table.table['date_occurred'].max()}")
```

--------------------------------

### Importing Libraries

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Initializes the OpenPoliceData and pandas libraries for data loading and manipulation.

```python
# This is a comment. It is not code and does not run

# Let's load in the libraries that we will use

# This loads in the Open Police Data package and renames it as opd (for convenience, the as opd
# could be left off and the package would be referred openpolicedata)
# OpenPoliceData is a package for loading in police datasets
import openpolicedata as opd
# Same deal except for the pandas package. pandas is a library for manipulating, analyzing, and plotting tabular data
import pandas as pd
```

--------------------------------

### Request year range for single multi-year dataset

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/date_filtering.ipynb.txt

Request a specific range of years for a single multi-year dataset by providing a list with the start and end year.

```python
table = northampton.load('USE OF FORCE', [2018,2020])
table
```

--------------------------------

### Initialize OpenPoliceData Source

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.ipynb.txt

Create a source object for Fairfax County to access available datasets and loading methods.

```python
import openpolicedata as opd
src = opd.Source(state="Virginia", agency="Fairfax County Police Department")
```

--------------------------------

### Inspect Table Data and Columns

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.ipynb.txt

Display the first few rows of a table and list all available column names.

```python
print(f"df_all is a {type(t.table)}")

# head() prints out the first 5 rows. If you want to print out a different # of rows (n) use head(n)
t.table.head()
```

```python
t.table.columns
```

--------------------------------

### Load Dataset by URL

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/date_filtering.ipynb.txt

Loads a specific dataset using the Asheville source and a URL from the dataset metadata.

```python
table = asheville.load('USE OF FORCE', "MULTIPLE", url=uof_datasets.iloc[0]['URL'])
table
```

--------------------------------

### Preview Loaded Data

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/export_dataset_to_csv_and_other_file_types.html

Display the first few rows of the loaded table to verify the data structure.

```python
# Show the first 5 rows of the table
t.table.head(n=5)
# Now you are ready for analyzing the data in the table t.
```

--------------------------------

### Inspect Available Table Types

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Retrieve a list of available table types for the initialized source.

```python
# Find out what types of data are available from this source
types = src.get_tables_types()

print(types)
```

--------------------------------

### Initialize Charge Association Analysis

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.ipynb.txt

Prepares to look up case numbers for associated charges in the Drunk in Public dataset.

```python
# This is going to be a little trickier. We need to look up the case numbers in our Drunk in Public DataFrame df_no_dups 

```

--------------------------------

### Load Police Dataset

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Initialize a data source and load a specific table type for a given year.

```python
# Load traffic stops data from New Orleans for the year 2022.
src = opd.Source("New Orleans")
tbl = src.load(table_type="STOPS", date=2022)
```

--------------------------------

### Inspect Dataset Columns

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/index.ipynb.txt

Initializes a source object to inspect the available columns in a dataset.

```python
src = opd.Source("Phoenix")
```

--------------------------------

### Import openpolicedata and dependencies

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analyze_datasets.html

Initial imports required for data loading and manipulation.

```python
import openpolicedata as opd
import pandas as pd
import numpy as np
```

--------------------------------

### Diagnose Dataset Loading Errors

Source: https://openpolicedata.readthedocs.io/en/stable/troubleshooting/index.html

Demonstrates how to identify HTTP errors when a dataset fails to load and how to verify the source URL in a browser.

```python
> src = opd.Source('Buffalo')
> # Attempt to load dataset that was removed by Buffalo (now also removed from OPD)
> src.load(table_type='TRAFFIC ARRESTS',date=2023)

openpolicedata.exceptions.OPD_SocrataHTTPError: ('data.buffalony.gov', '5kqt-m62h', '404 Client Error: Not Found', 'There is likely an issue with the website. Open the URL https://data.buffalony.gov/resource/5kqt-m62h.json with a web browser to confirm. See a list of known site outages at https://github.com/openpolicedata/opd-data/blob/main/outages.csv')
```

```python
> src = opd.Source('Tucson')
> idx = (src.datasets['TableType']=='ARRESTS') &  (src.datasets['Year']==2017)  # Find data
> src.datasets[idx]['URL'].iloc[0]  
'https://services3.arcgis.com/9coHY2fvuFjG9HQX/arcgis/rest/services/OpenData_PublicSafety/FeatureServer/0'
> src.datasets[idx]['source_url'].iloc[0]
'https://gisdata.tucsonaz.gov/datasets/cotgis::tucson-police-arrests-2017-open-data/about'
```

--------------------------------

### Check available data types and years

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Retrieves the supported table types and years for the initialized source.

```python
print(src.get_tables_types())
print(src.get_years(table_type="TRAFFIC STOPS"))
```

--------------------------------

### Initialize Default Known Columns

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Retrieve the default column mapping for a specific dataset to serve as a base for manual overrides.

```python
src = opd.Source("Phoenix")
tbl = src.load(table_type="USE OF FORCE", date=2022, pbar=False)  # pbar=False does not show progress bar
default_known_cols = {opd.Column.DATE:tbl.date_field, opd.Column.AGENCY:tbl.agency_field}
default_known_cols
```

--------------------------------

### Access Standardized Column Definitions

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Demonstrates how to access the opd.Column definitions to identify standardized column names used during the standardization process.

```python
wrap_print("Attribute value below is an attribute of opd.defs.columns (i.e. opd.defs.columns.AGE_OFFICER) that can be used to access column names")
wrap_print('Column Name value below is the corresponding standardized column name (i.e. "OFFICER_AGE") that replaces the original column name '+ 
           'during the standardization process')

opd.Column
```

--------------------------------

### Load Phoenix Use of Force Dataset

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Initializes a source object and loads the 2022 Use of Force dataset for Phoenix to inspect raw column names.

```python
src = opd.Source("Phoenix")
tbl = src.load(table_type="USE OF FORCE", date=2022, pbar=False)  # pbar=False does not show progress bar
# Only showing 1st 30 due to large number of columns
tbl.table.columns[:30]
```

--------------------------------

### Inspect Standardized Data

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Display the first few rows of the table after standardization has been applied.

```python
tbl.table.head()
```

--------------------------------

### Define Output Directory for Export

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/export_dataset_to_csv_and_other_file_types.html

Sets the current working directory as the destination for exported files.

```python
import os
cwd = os.getcwd()
csv_filepath = cwd

print(f"The CSV file will be written to {csv_filepath}. Make sure this path is okay before running the next cell. If the path is not okay then modify csv_filepath.")
```

--------------------------------

### Filter and Compare Arrest Data with Python

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Filters arrest records by frequency threshold and calculates demographic distributions for repeat offenders compared to the total dataset.

```python
  
threshold = 2  # This could be varied to evaluate different cases

vc = df_single_charge["Arrestee Name ID"].value_counts()
# Remove value counts that do not exceed the threshold
vc_exceed_thresh = vc[vc>=threshold]

# The row labels are termed indices and can be accessed with .index
name_ids_for_exceeds = vc_exceed_thresh.index

# Let's create a DataFrame that only contains the names in name_ids_for_exceeds

# .isin(name_ids_for_exceeds) is True for values that are in name_ids_for_exceeds
has_matching_name = df_single_charge["Arrestee Name ID"].isin(name_ids_for_exceeds)

# Let's only keep one row per name so that each is only counted once when we look at race/ethnicity
df_exceeds = df_single_charge[has_matching_name].drop_duplicates(subset="Arrestee Name ID")

print(f"Counts for all cases (Total: {len(df_single_charge)}):")
print(df_single_charge["race_eth"].value_counts())

print(f"\n\nCounts for individuals arrests multiple times (Total: {len(df_exceeds)}):")
print(df_exceeds["race_eth"].value_counts())

print(f"\n\nPercentages for all cases (Total: {len(df_single_charge)}):")
print(df_single_charge["race_eth"].value_counts(normalize=True).round(3)*100)

print(f"\n\nPercentages for individuals arrests multiple times (Total: {len(df_exceeds)}):")
print(df_exceeds["race_eth"].value_counts(normalize=True).round(3)*100)
```

--------------------------------

### List Available Table Types

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Retrieve the types of tables available for a specific source.

```python
src.get_tables_types()
```

--------------------------------

### Query available datasets

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/index.ipynb.txt

Retrieves a list of all datasets available in the OpenPoliceData library.

```python
import openpolicedata as opd
opd.datasets.query()
```

--------------------------------

### Query All Datasets

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/finding_datasets.html

Retrieves the complete table of available datasets as a pandas DataFrame.

```python
# Query for the entire table of available data as a pandas DataFrame (https://pandas.pydata.org/docs/user_guide/10min.html#min)
# This shows all the datasets that are available for access
# This information can be filtered to find a dataset of interest
datasets = opd.datasets.query()
datasets.head()
```

--------------------------------

### Load Dataset and Inspect Columns

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/date_filtering.ipynb.txt

Loads the Chicago stops dataset and displays the available columns to identify the date field.

```python
table = chicago.load('STOPS', 'MULTIPLE')
table.table.columns
```

--------------------------------

### Standardize Complex Datasets

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Demonstrates standardization where some values map to defaults and others remain unchanged.

```python
src = opd.Source("California")
tbl = src.load(opd.TableType.DEATHS_IN_CUSTODY, "MULTIPLE")
tbl.standardize()
s = tbl.get_transform_map(new=opd.Column.RACE_ETHNICITY_SUBJECT)

print(f"orig_column_name: {s.orig_column_name}")
print(f"new_column_name: {s.new_column_name}\n")
print("Data Transformation Map:")
s.data_maps
```

```python
{'White': 'WHITE',
 'Hispanic': 'HISPANIC/LATINO',
 'Black': 'BLACK',
 'Other': 'OTHER',
 'American Indian': 'INDIGENOUS',
 'Other Asian': 'ASIAN',
 'Filipino': 'Filipino',
 'Vietnamese': 'Vietnamese',
 'Pacific Islander': 'ASIAN/PACIFIC ISLANDER',
 'Asian Indian': 'Asian Indian',
 'Chinese': 'Chinese',
 'Korean': 'Korean',
 'Laotian': 'Laotian',
 'Samoan': 'Samoan',
 'Cambodian': 'Cambodian',
 'Japanese': 'Japanese',
 'Hawaiian': 'ASIAN/PACIFIC ISLANDER',
 'Guamanian': 'Guamanian',
 <NA>: 'UNSPECIFIED'}
```

--------------------------------

### Summarize datasets by state and year

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/summarize_opd_data.ipynb.txt

Generates a summary of the number of datasets per state, categorized by year.

```python
opd.datasets.summary_by_state(by="year").head(7)
```

--------------------------------

### Querying Traffic Stop Datasets

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Search for traffic stop datasets in Maryland and display the results.

```python
# We will load Montgormery County, Maryland traffic stop data. First show our dataset options.
df = opd.datasets.query(table_type='TRAFFIC STOPS', state="Maryland")
df.head()
```

--------------------------------

### Query and Filter Datasets

Source: https://openpolicedata.readthedocs.io/en/stable/datasets/index.html

Retrieves the full list of datasets and filters the output to display specific columns.

```python
opd.datasets.query().loc[:, ["State", "SourceName", "Agency", "AgencyFull", "TableType", "coverage_start", "coverage_end"]]
```

--------------------------------

### Calculate percentage of race and ethnicity

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Convert normalized ratios into percentages.

```python
# Or if you'd rather have the value as a percent just multiply by 100
df["race_eth"].value_counts(normalize=True)*100
```

--------------------------------

### Upgrade OpenPoliceData

Source: https://openpolicedata.readthedocs.io/en/stable/documentation.html

Command to update the OpenPoliceData library to the latest version via pip.

```bash
pip install openpolicedata --upgrade
```

--------------------------------

### Handle Deprecation Warnings

Source: https://openpolicedata.readthedocs.io/en/stable/troubleshooting/index.html

Shows how to identify and update code when encountering a DeprecationWarning for outdated function usage.

```python
> src = opd.Source('Oakland')
> tbl_dep = src.load_from_url(2023, 'STOPS')
DeprecationWarning: load_from_url is deprecated and will be removed in a future release. Please use load instead. load uses the same inputs except table_type now comes before year.
> tbl = src.load('STOPS', 2023)
> tbl.table.equals(tbl_dep.table)
True
```

--------------------------------

### Requesting an entire multi-year dataset

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/date_filtering.ipynb.txt

Pass 'MULTIPLE' as the date argument to retrieve the full multi-year dataset.

```python
table = northampton.load('USE OF FORCE', "MULTIPLE")
table
```

--------------------------------

### Summarize Datasets by State and Year

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/summarize_opd_data.html

Retrieve a summary of datasets grouped by state and broken down by year.

```python
# Find number of datasets from each state broken down by year
opd.datasets.summary_by_state(by="year").head(7)
```

--------------------------------

### Calculate Race/Ethnicity Percentages for Unique Individuals

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Compares race/ethnicity distributions between the full dataset and a subset containing only unique individuals.

```python
# Let's look at only counting each individual once first
df_each_ind_once = df_no_dups.drop_duplicates(subset="Arrestee Name ID")

print(f"Percentages for all cases (Total: {len(df_no_dups)}):")
print(df_no_dups["race_eth"].value_counts(normalize=True).round(3)*100)

print(f"\n\nPercentages for unique individuals (Total: {len(df_each_ind_once)}):")
print(df_each_ind_once["race_eth"].value_counts(normalize=True).round(3)*100)
```

--------------------------------

### Accessing Column Definitions

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/standardization.ipynb.txt

Displays the mapping between attribute names and standardized column names used in the library.

```python
wrap_print("Attribute value below is an attribute of opd.defs.columns (i.e. opd.defs.columns.AGE_OFFICER) that can be used to access column names")
wrap_print('Column Name value below is the corresponding standardized column name (i.e. "OFFICER_AGE") that replaces the original column name '+
           'during the standardization process')

opd.Column
```

--------------------------------

### Customize Gender Category Labels

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Demonstrates how to modify the gender category dictionary and apply it during the standardization of a loaded dataset.

```python
new_gender_cats = gender_cats.copy()
new_gender_cats["MALE"] = "M"

src = opd.Source("Berkeley")
tbl = src.load(opd.defs.TableType.STOPS, 2022)
tbl.standardize(gender_cats=new_gender_cats)


wrap_print(f"Unique gender values are {tbl.table[opd.Column.GENDER_SUBJECT].unique()}")
wrap_print(f"In previous examples, raw values for Male were replaced with '{gender_cats['MALE']}'. "+
      f"In this case, customization resulted in replacement with '{new_gender_cats['MALE']}'.")
```

--------------------------------

### Load Dataset

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Load a specific dataset into a table object for a given year.

```python
tbl = src.load("FIELD CONTACTS", 2022)
tbl
```

--------------------------------

### Load Dataset

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Load a specific dataset into a variable by providing the year and table type.

```python
# Load traffic stop data for 2021
t = src.load(date=2021, table_type='TRAFFIC STOPS')
```

--------------------------------

### Export Data to CSV

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/export_dataset_to_csv_and_other_file_types.html

Exports the dataset to a CSV file using the to_csv method. Similar methods exist for Feather and Parquet formats.

```python
# Save to CSV. To specify a custom filename, set the filename input
csv_written_filename=t.to_csv(output_dir=csv_filepath)
print(f"The CSV file was written to {csv_written_filename}.")
```

--------------------------------

### Summarize datasets by table type

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/summarize_opd_data.ipynb.txt

Generates a summary of the number of datasets available for each table type.

```python
opd.datasets.summary_by_table_type()
```

--------------------------------

### Load multi-year dataset by ID

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/date_filtering.ipynb.txt

Load a specific multi-year dataset by identifying its ID from the dataset list.

```python
ids = mesa.datasets.loc[mesa.datasets['TableType']=='CALLS FOR SERVICE','dataset_id']
table = mesa.load('CALLS FOR SERVICE', "MULTIPLE", id=ids.iloc[1], nrows=1000)
table
```

--------------------------------

### Save and load local data

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Save tables to local files and reload them using specific source methods.

```python
tbl.to_csv()  # Can also call to_feather and to_parquet with the same inputs
new_src = opd.Source("Columbia", state="South Carolina")
new_tbl = new_src.load_csv("FIELD CONTACTS", 2022)  # Can also call load_feather and load_parquet with the same inputs
new_tbl.table.head(2)
```

--------------------------------

### Inspect Standardized Columns

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Displays the first and last columns of the table after standardization to verify the structure.

```python
print(f"The 1st 10 columns after standardization are: {tbl.table.columns[:12]}")
print(f"The last 11 columns after standardization are: {tbl.table.columns[-9:]}")
```

--------------------------------

### Display cleaned table

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Show the final processed table.

```python
# Now let's re-display the table
df_sizes
```

--------------------------------

### Query datasets by state

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/finding_datasets.ipynb.txt

Retrieve all available datasets for a specific state using the query method.

```python
df = opd.datasets.query(state="Maryland")
df.head()
```

--------------------------------

### Load Single Year Dataset

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/date_filtering.ipynb.txt

Load a specific year of data by providing the year as an argument to the load method.

```python
table = northampton.load('USE OF FORCE', 2021)
```

--------------------------------

### Compare Arrest Percentages

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Calculates and prints the distribution of arrest types by race before and after removing duplicates.

```python
# Let's look at arrest types by race/ethnicity for our table with duplicates dropped

# Same operation previously done across multiple cells rolled up into a single line of code
df_sizes_no_dups = df_no_dups.groupby(["race_eth","Arrest Type"]).size().unstack().fillna(0).convert_dtypes()

print("Percentages BEFORE duplicates dropped:")
# Putting the below in parentheses allows calling functions on the resulting DataFrame after multiplication
# .round(1) rounds to 1 decimal place
print((df_sizes.divide(df_sizes.sum(axis=1), axis=0) * 100).round(1))

print("\n\nPercentages AFTER duplicates dropped:")
print((df_sizes_no_dups.divide(df_sizes_no_dups.sum(axis=1), axis=0) * 100).round(1))
```

--------------------------------

### Summarize Datasets by Table Type

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/summarize_opd_data.html

Retrieve a summary of the number of datasets available for each type of table.

```python
# Find number of datasets for each type of table
opd.datasets.summary_by_table_type()
```

--------------------------------

### Inspect Data Transformation Map

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Loads a dataset and inspects the mapping between original column values and standardized categories.

```python
src = opd.Source("Virginia")
tbl = src.load("STOPS", 2022,agency="Norfolk Police Department")
tbl.standardize()
s = tbl.get_transform_map(new=opd.Column.RACE_SUBJECT)

print(f"orig_column_name: {s.orig_column_name}")
print(f"new_column_name: {s.new_column_name}\n")
print("Data Transformation Map:")
s.data_maps
```

```python
{'BLACK OR AFRICAN AMERICAN': 'BLACK',
 'WHITE': 'WHITE',
 'UNKNOWN': 'UNKNOWN',
 'ASIAN OR NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER': 'ASIAN/PACIFIC ISLANDER',
 'AMERICAN INDIAN OR ALASKA NATIVE': 'INDIGENOUS'}
```

--------------------------------

### Use Compact Gender Categories

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Applies abbreviated gender labels by passing 'compact' to the standardize method or retrieving compact definitions.

```python
compact_gender_cats = opd.defs.get_gender_cats(compact=True)

src = opd.Source("Berkeley")
tbl = src.load(opd.defs.TableType.STOPS, 2022)
tbl.standardize(gender_cats='compact')


wrap_print(f"Unique gender values are {tbl.table[opd.Column.GENDER_SUBJECT].unique()}")
wrap_print(f"In previous examples, raw values for Female were replaced with '{gender_cats['FEMALE']}'. "+
      f"In this case, customization resulted in replacement with '{compact_gender_cats['FEMALE']}'.")
```

--------------------------------

### Inspect Loaded Dataset

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Access the underlying pandas DataFrame from the table object to view the first 5 rows of the dataset.

```python
# The loaded table is stored in the table parameter as a pandas DataFrame (https://pandas.pydata.org/docs/user_guide/10min.html#min)
# Show the first 5 rows of the table
t.table.head(n=5)
# Now you are ready for analyzing the data in the table t.
```

--------------------------------

### Load specific dataset via URL substring

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/date_filtering.html

Use a URL substring to specify which dataset to load when multiple datasets contain the requested year.

```python
table = northampton.load('USE OF FORCE', 2021, url='2014-2020')
table
```

```text
state: Massachusetts,
source_name: Northampton,
agency: Northampton,
table_type: USE OF FORCE,
year: MULTIPLE,
description: <NA>,
url: https://northamptonpd.com/images/ODP%20Spreadsheets/NPD_Use_of_Force_2014-2020_incident_level_data.xlsx,
source_url: https://northamptonpd.com/open-data-portal.html,
urls: {'source_url': 'https://northamptonpd.com/open-data-portal.html', 'readme': None, 'data': 'https://northamptonpd.com/images/ODP%20Spreadsheets/NPD_Use_of_Force_2014-2020_incident_level_data.xlsx'}
```

--------------------------------

### Standardize a Table

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Load a dataset and apply standardization to normalize column names and values.

```python
src = opd.Source("Virginia")
tbl = src.load(table_type="STOPS", date=2022, agency="Arlington County Police Department")
tbl.standardize()

print(f'The columns after standardization are: \n{tbl.table.columns}\n')
```

--------------------------------

### Create New DataFrame Column

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.ipynb.txt

Initialize a new column in the DataFrame for derived data.

```python
# Now, let's create a new column in our date frame for combined race ethnicity

# df_all[["Arrestee Race","Arrestee Ethnicity"]] extracts just the race and ethnicity columns into a temporary new DataFrame
# This is purely for efficiency reasons and is not necessary
```

--------------------------------

### Load and Inspect Police Data

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/getting_started/index.ipynb.txt

Loads a specific dataset and displays the first 30 columns for inspection.

```python
tbl = src.load(table_type="USE OF FORCE", date=2022, pbar=False)  # pbar=False does not show progress bar
# Only showing 1st 30 due to large number of columns
tbl.table.columns[:30]
```

--------------------------------

### Retrieve Standardization Map

Source: https://openpolicedata.readthedocs.io/en/stable/getting_started/index.html

Retrieves and prints the transformation map showing column renames and value mappings.

```python
std_map = tbl.get_transform_map(minimize=True)
for t in std_map:
    print(f"{t}\n")
```

--------------------------------

### Retrieve Compact Race Categories

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/standardization.html

Retrieves the mapping dictionary for compact race categories.

```python
opd.defs.get_race_cats(compact=True)
```

--------------------------------

### Calculate and Print Arrest Statistics

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.html

Groups arrest data by race and type to calculate counts and percentages for both total and single-charge arrests.

```python
df_single_charge = df_no_dups[matches_single_charge]
df_sizes_single_charge = df_single_charge.groupby(["race_eth","Arrest Type"]).size().unstack().fillna(0).convert_dtypes()

print("Counts for all arrests for Drunk in Public:")
print(df_sizes_no_dups)

print("\n\nCounts for single-charge arrests for Drunk in Public:")
print(df_sizes_single_charge)

print("\n\nPercentages for all arrests for Drunk in Public:")
print((df_sizes_no_dups.divide(df_sizes_no_dups.sum(axis=1), axis=0) * 100).round(1))

print("\n\nPercentages for single-charge arrests for Drunk in Public:")
print((df_sizes_single_charge.divide(df_sizes_single_charge.sum(axis=1), axis=0) * 100).round(1))
```

--------------------------------

### Summarize datasets by table type

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/summarize_opd_data.ipynb.txt

Generates a summary of datasets grouped by table type, optionally broken down by year.

```python
# Find number of datasets for each type of table broken down by year
opd.datasets.summary_by_table_type(by_year=True).head()
```

```python
# All returned summary tables are pandas DataFrames so they can be easily exported to CSV files using pandas to_csv.
# Find number of datasets for each type of table broken down by year
opd.datasets.summary_by_table_type(by_year=True).to_csv("table_summary_by_year.csv")
```

--------------------------------

### Check Available Years

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Query the available years for a specific table type. Note that requests without an app_token may be subject to throttling.

```python
# Find out what years are available from the stops table
# IF you do not have a key setup you may see the message: "WARNING:root:Requests made without an app_token will be subject to strict throttling limits." This is normal.
years = src.get_years(table_type=types[0])
print(years)
```

--------------------------------

### Load and Inspect Arrest Data

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/analysis_examples/example_arrests_analysis_with_python_tutorial.ipynb.txt

Loads arrest data for a specified year and table type, then prepares it for analysis using pandas.

```python
# date and table_type should match values in src.datasets
t = src.load(date=2022, table_type="ARRESTS")

# You can check the type of something with the type function
# table is a table of data that was just loaded. 
# The table is a DataFrame from the pandas Python package. Any of the techniques applied to t.table can be applied to other pandas DataFrames
# The below print statement prints out the text in the string except it replaces {type(t.table)} with the type of df_all
```

--------------------------------

### Load dataset

Source: https://openpolicedata.readthedocs.io/en/stable/_sources/examples/opd-examples/loading_datasets.ipynb.txt

Loads the specific table type for a given year into a dataframe.

```python
data = src.load(table_type="TRAFFIC STOPS", year=2023)
data.table.head()
```

--------------------------------

### Summarize datasets by table type

Source: https://openpolicedata.readthedocs.io/en/stable/examples/opd-examples/summarize_opd_data.html

Generates a summary of datasets categorized by table type, with an option to break down the counts by year.

```python
# Find number of datasets for each type of table broken down by year
opd.datasets.summary_by_table_type(by_year=True).head()
```
Tokens:
Raw