Open States

https://docs.openstates.org/
Admin
Open States aggregates, standardizes, and publishes legislative information and data from all 50...
Tokens:36,515
Snippets:299
Trust Score:10
Update:4 weeks ago
Show doc for...
Tokens:
Raw
### Install dependencies with poetry

Source: https://docs.openstates.org/contributing/people

Installs project dependencies using poetry. The output shows the installation process.

```bash
$ poetry install
Installing dependencies from lock file
...


```

--------------------------------

### Start the database container

Source: https://docs.openstates.org/contributing/scrapers

Run `docker-compose up -d db` from the `openstates.org` directory to start the database container if it is not running.

```bash
$ docker-compose up -d db

```

--------------------------------

### Install pre-commit hooks

Source: https://docs.openstates.org/contributing/openstates-org

Installs pre-commit hooks for the repository. The output confirms the hook was installed at .git/hooks/pre-commit.

```bash
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

```

--------------------------------

### Start the Database with docker-compose up -d db

Source: https://docs.openstates.org/contributing/local-database

Starts the database container. If already running, output shows 'scrapers_db_1 is up-to-date'.

```bash
openstates-scrapers$ docker-compose up -d db
Starting scrapers_db_1 ... done

```

```bash
openstates-scrapers$ docker-compose up -d db
scrapers_db_1 is up-to-date

```

--------------------------------

### Install Poetry via curl

Source: https://docs.openstates.org/contributing

Official Poetry installation command. Run this in your terminal to install Poetry, which manages the Python virtual environment for the project.

```bash
curl -sSL https://install.python-poetry.org | python3 -
```

--------------------------------

### Basic Event Info

Source: https://docs.openstates.org/data/query-scraper-output-data

Selects basic event fields including start date, all-day flag, name, location name, status, classification, and description, ordered by start date.

```sql
SELECT start_date, all_day, name, location.name, status, classification, description
FROM events
ORDER BY start_date;

```

--------------------------------

### Reset the Database with scripts/init-db.sh

Source: https://docs.openstates.org/contributing/local-database

Resets the database to scratch. Useful for clearing bad data or starting fresh.

```bash
openstates-scrapers/$ ./scripts/init-db.sh

```

--------------------------------

### Check Docker and docker-compose versions

Source: https://docs.openstates.org/contributing

Verify that Docker and docker-compose are installed and check their versions. Ensure they are relatively recent to avoid issues.

```bash
$ docker --version
Docker version 19.03.4, build 9013bf5
$ docker-compose --version
docker-compose version 1.24.1, build 4667896b
```

--------------------------------

### Event Info with Source URL

Source: https://docs.openstates.org/data/query-scraper-output-data

Joins events with event_sources to include the source URL, ordered by start date.

```sql
SELECT e.start_date, e.all_day, e.name, e.location.name, e.status, e.classification, e.description, s.url
FROM events e
LEFT JOIN event_sources s ON e._id = s.event_id
ORDER BY start_date;

```

--------------------------------

### Example log output from a successful scrape

Source: https://docs.openstates.org/contributing/scrapers

This is the expected log output when running the scraper. It shows the fetching of member list and biography pages. If a scrape fails, it's likely an issue with the scraper.

```text
INFO:scrapers_next.nc.people.SenList:fetching https://www.ncleg.gov/Members/MemberTable/S
INFO:scrapers_next.nc.people.LegDetail:fetching https://www.ncleg.gov/Members/Biography/S/430
INFO:scrapers_next.nc.people.LegDetail:fetching https://www.ncleg.gov/Members/Biography/S/431
INFO:scrapers_next.nc.people.LegDetail:fetching https://www.ncleg.gov/Members/Biography/S/432
INFO:scrapers_next.nc.people.LegDetail:fetching https://www.ncleg.gov/Members/Biography/S/433
INFO:scrapers_next.nc.people.LegDetail:fetching https://www.ncleg.gov/Members/Biography/S/434

```

--------------------------------

### Start DuckDB and query Bills JSON files

Source: https://docs.openstates.org/data/query-scraper-output-data

Launch DuckDB in the scraper output directory and run a SQL query to read all bill JSON files. The output shows DuckDB version and a prompt.

```sql
jesse@work:~/repo/openstates/openstates-scrapers/scrapers/_data/de$ duckdb
v0.10.2 1601d94f94
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D SELECT * FROM read_json('bill_*.json');
```

--------------------------------

### Run a spatula scraper with poetry run spatula scrape

Source: https://docs.openstates.org/contributing/writing-a-committee-scraper

Use this command to run a spatula scraper. Replace {directory your python file is in}.{state your scraper is for}.committees.{function (optional)} with the appropriate module path. The example shows running the entire committee scraper for Missouri.

```bash
poetry run spatula scrape {directory your python file is in}.{state your scraper is for}.committees.{function (optional)}
```

```bash
poetry run spatula scrape scrapers_next.mo.committees
```

--------------------------------

### Preferred intentionally fragile scraper example

Source: https://docs.openstates.org/contributing/testing-scrapers

Uses a dictionary lookup that raises a KeyError when an unknown party abbreviation is encountered, forcing immediate attention and a fix. This is the preferred approach.

```python
party_abbr = doc.xpath('//span[@class="partyabbr"])
party = {'D': 'Democratic', 'R': 'Republican'}[party_abbr]

```

--------------------------------

### Example of complete committee data (JSON)

Source: https://docs.openstates.org/contributing/writing-a-committee-scraper

Shows the expected JSON structure for a committee, including name, chamber, classification, parent, sources, links, other_names, members, and extras. Use this as a reference for the data your scraper must produce.

```json
{
"name": "Revenue", 
"chamber": "upper",
"classification": "committee", 
"parent": null, 
"sources": [
    {"url": "https://www.ilga.gov/senate/committees/default.asp", "note": "homepage"}, 
    {"url": "https://www.ilga.gov/senate/committees/members.asp?CommitteeID=2688", "note": ""}], 
"links": [], 
"other_names": [], 
"members": [
    {"name": "Mattie Hunter", "role": "Chair", "person_id": null}, 
    {"name": "Steve Stadelman", "role": "Vice-Chair", "person_id": null}, 
    {"name": "Robert F. Martwick", "role": "Member", "person_id": null}, 
    {"name": "Cristina H. Pacione-Zayas", "role": "Member", "person_id": null}, 
    {"name": "Robert Peters", "role": "Member", "person_id": null}, 
    {"name": "Elgie R. Sims, Jr.", "role": "Member", "person_id": null}, 
    {"name": "Donald P. DeWitte", "role": "Minority", "person_id": null}, 
    {"name": "Dale Fowler", "role": "Member", "person_id": null}, 
    {"name": "Jil Tracy", "role": "Member", "person_id": null}, 
    {"name": "Sally J. Turner", "role": "Member", "person_id": null}], 
"extras": {}
}
```

--------------------------------

### Scraping log output

Source: https://docs.openstates.org/contributing/scrapers

Example log output during the actual data scraping, showing GET requests and bill saves for the 2017 session.

```text
08:46:36 INFO openstates: no session specified, using 2017
08:46:36 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/SimpleBillInquiry/displaybills.pl?Session=2017&tab=Chamber&Chamber=Senate
08:46:38 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/BillLookUp/BillLookUp.pl?Session=2017&BillID=S1
08:46:39 INFO openstates: save bill SR 1 in 2017 as bill_03c7edb4-72d2-11e7-8df8-0242ac130003.json
08:46:39 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/BillLookUp/BillLookUp.pl?Session=2017&BillID=S2
08:46:39 INFO openstates: save bill SJR 2 in 2017 as bill_044a5fc4-72d2-11e7-8df8-0242ac130003.json
08:46:39 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/BillLookUp/BillLookUp.pl?Session=2017&BillID=S3
08:46:40 INFO openstates: save bill SB 3 in 2017 as bill_04e8c66e-72d2-11e7-8df8-0242ac130003.json
08:46:40 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/BillLookUp/BillLookUp.pl?Session=2017&BillID=S4
08:46:41 INFO openstates: save bill SB 4 in 2017 as bill_05781f08-72d2-11e7-8df8-0242ac130003.json
08:46:41 INFO scrapelib: GET - http://www.ncga.state.nc.us/gascripts/BillLookUp/BillLookUp.pl?Session=2017&BillID=S5

```

--------------------------------

### Initialize Database for OpenStates.org with docker/init-db.sh

Source: https://docs.openstates.org/contributing/local-database

Run from the openstates.org directory after running openstates-scrapers' init-db.sh. Creates Django tables and a local API key 'testkey'.

```bash
openstates.org/$ ./docker/init-db.sh
+ unset DATABASE_URL
+ docker-compose run --rm -e PYTHONPATH=docker/ --entrypoint 'poetry run ./manage.py migrate' django
Creating openstatesorg_django_run ... done
Operations to perform:
  Apply all migrations: account, admin, auth, bulk, bundles, contenttypes, dashboards, data, people_admin, profiles, sessions, sites, social
Running migrations:
  Applying auth.0001_initial... OK

...TRUNCATED...

docker-compose run --rm -e PYTHONPATH=docker/ --entrypoint 'poetry run ./manage.py shell -c "import testdata"' django

```

--------------------------------

### Show os-people command help

Source: https://docs.openstates.org/contributing/people

Displays help for the os-people command, which manages people data.

```bash
poetry run os-people --help

```

--------------------------------

### Coarse citation example

Source: https://docs.openstates.org/enhancement-proposals/002-legal-citations

Example of a legal citation at a coarse granularity, referencing a whole section of Minnesota Statutes.

```text
"Minnesota Statutes 2018, section 31"

```

--------------------------------

### Initialize Database for Scraping with scripts/init-db.sh

Source: https://docs.openstates.org/contributing/local-database

Run from the openstates-scrapers directory. Resets the database to scratch if run again. Populates tables and static data for scraping.

```bash
openstates-scrapers/$ ./scripts/init-db.sh
+ unset DATABASE_URL
+ docker-compose down
Removing scrapers_db_1 ... done
Removing network openstates-network
+ docker volume rm openstates-postgres
openstates-postgres
+ docker-compose up -d db
Creating network "openstates-network" with the default driver
Creating volume "openstates-postgres" with default driver
Creating scrapers_db_1 ... done
+ sleep 3
+ DATABASE_URL=postgis://openstates:openstates@db/openstatesorg
+ docker-compose run --rm --entrypoint 'poetry run os-initdb' scrape
Creating scrapers_scrape_run ... done
Operations to perform:
  Apply all migrations: contenttypes, data
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying data.0001_initial... OK
  Applying data.0002_auto_20200422_0028... OK
  Applying data.0003_auto_20200422_0031... OK

...TRUNCATED...

loading WY
loading DC
loading PR
loading US

```

--------------------------------

### Run sample command for North Carolina

Source: https://docs.openstates.org/contributing/text-extraction

Runs the sample subcommand against North Carolina (nc) using docker-compose. The output lists each processed PDF and the extracted text file with byte counts, ending with a summary line.

```bash
$ docker-compose run --rm text-extract sample nc
raw/nc/2017-HR 924-Edition 1.pdf => text/nc/2017-HR 924-Edition 1.pdf.txt (1507 bytes)
raw/nc/2017-HB 1034-Edition 1.pdf => text/nc/2017-HB 1034-Edition 1.pdf.txt (3096 bytes)
raw/nc/2019-SB 421-Edition 1.pdf => text/nc/2019-SB 421-Edition 1.pdf.txt (961 bytes)
raw/nc/2019-HB 430-Edition 1.pdf => text/nc/2019-HB 430-Edition 1.pdf.txt (4831 bytes)
raw/nc/2017-SB 753-Edition 1.pdf => text/nc/2017-SB 753-Edition 1.pdf.txt (719 bytes)
raw/nc/2019-HB 788-Edition 1.pdf => text/nc/2019-HB 788-Edition 1.pdf.txt (2674 bytes)
raw/nc/2017-SB 373-Filed.pdf => text/nc/2017-SB 373-Filed.pdf.txt (18538 bytes)
raw/nc/2019-SB 574-Filed.pdf => text/nc/2019-SB 574-Filed.pdf.txt (1712 bytes)
raw/nc/2017-SJR 686-Resolution 2017-12.pdf => text/nc/2017-SJR 686-Resolution 2017-12.pdf.txt (15928 bytes)
raw/nc/2017-HB 1007-Filed.pdf => text/nc/2017-HB 1007-Filed.pdf.txt (6248 bytes)
nc: processed 10, 0 skipped, 0 missing, 0 empty

```

--------------------------------

### Fine-grained citation examples

Source: https://docs.openstates.org/enhancement-proposals/002-legal-citations

Example of legal citations at a finer granularity, listing specific subsections and ranges. This level of detail typically requires a full-featured legal citation parser and processing of bill texts.

```text
"Minnesota Statutes 2018, section 31A.10",
"Minnesota Statutes 2018, section 31A.14",
"Minnesota Statutes 2018, section 31B.10-12"

```

--------------------------------

### Vote Events General Summary

Source: https://docs.openstates.org/data/query-scraper-output-data

Selects key fields from vote_events, including bill identifier, start date, motion text, first two counts, and first source URL, ordered by bill and start date.

```sql
SELECT bill_identifier AS bill, start_date, motion_text,
counts[1].option AS opt1, counts[1].value AS opt1_num,
counts[2].option AS opt2, counts[2].value AS opt2_num,
sources[1].url AS source_url
FROM vote_events
ORDER BY bill_identifier, start_date;

```

--------------------------------

### GET /committees

Source: https://docs.openstates.org/api-v3

Retrieves a list of committees for a specified jurisdiction.

```APIDOC
## GET /committees

### Description
Get list of committees by jurisdiction.

### Method
GET

### Endpoint
https://v3.openstates.org/committees

### Parameters
#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)
- **jurisdiction** (string) - Required - Filter by jurisdiction

### Request Example
GET https://v3.openstates.org/committees?jurisdiction=ocd-jurisdiction/country:us/state:ca/government&apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **results** (array) - List of committees

#### Response Example
{
  "results": [
    {
      "id": "ocd-committee/1234",
      "name": "Committee on Finance"
    }
  ]
}
```

--------------------------------

### Run the import command

Source: https://docs.openstates.org/contributing/scrapers

Use `docker-compose run --rm scrape fl bills --fast` to import scraped data. The output shows the import progress and a summary of new, updated, and unchanged objects.

```bash
$ docker-compose run --rm scrape fl bills --fast
... (truncated) ...
23:03:34 ERROR openstates: cannot resolve pseudo id to Person: ~{"name": "Grant, M."}
23:03:36 ERROR openstates: cannot resolve pseudo id to Person: ~{"name": "Rodrigues, R."}
fl (import)
  bills: {}
import:
  bill: 0 new 0 updated 2620 noop
  jurisdiction: 0 new 0 updated 1 noop
  vote_event: 21 new 12 updated 533 noop

```

--------------------------------

### GET /people

Source: https://docs.openstates.org/api-v3

Lists or searches for people such as legislators and governors.

```APIDOC
## GET /people

### Description
List or search people (legislators, governors, etc.)

### Method
GET

### Endpoint
https://v3.openstates.org/people

### Parameters
#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)
- **jurisdiction** (string) - Optional - Filter by jurisdiction
- **name** (string) - Optional - Search by name

### Request Example
GET https://v3.openstates.org/people?jurisdiction=ocd-jurisdiction/country:us/state:ca/government&apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **results** (array) - List of people

#### Response Example
{
  "results": [
    {
      "id": "ocd-person/1234",
      "name": "Jane Doe",
      "roles": []
    }
  ]
}
```

--------------------------------

### GET /events

Source: https://docs.openstates.org/api-v3

Retrieves a list of events for a specified jurisdiction.

```APIDOC
## GET /events

### Description
Get list of events by jurisdiction.

### Method
GET

### Endpoint
https://v3.openstates.org/events

### Parameters
#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)
- **jurisdiction** (string) - Required - Filter by jurisdiction

### Request Example
GET https://v3.openstates.org/events?jurisdiction=ocd-jurisdiction/country:us/state:ca/government&apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **results** (array) - List of events

#### Response Example
{
  "results": [
    {
      "id": "ocd-event/1234",
      "name": "Committee Hearing"
    }
  ]
}
```

--------------------------------

### Run all tests

Source: https://docs.openstates.org/contributing/openstates-org

Runs the full test suite for the project using the provided script.

```bash
./docker/run-tests.sh

```

--------------------------------

### GET /jurisdictions

Source: https://docs.openstates.org/api-v3

Retrieves a list of all available jurisdictions in the Open States API.

```APIDOC
## GET /jurisdictions

### Description
Get a list of available jurisdictions.

### Method
GET

### Endpoint
https://v3.openstates.org/jurisdictions

### Parameters
#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)

### Request Example
GET https://v3.openstates.org/jurisdictions?apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **results** (array) - List of jurisdictions

#### Response Example
{
  "results": [
    {
      "id": "ocd-jurisdiction/country:us/state:ca/government",
      "name": "California"
    }
  ]
}
```

--------------------------------

### GET /people.geo

Source: https://docs.openstates.org/api-v3

Retrieves legislators for a given geographic location using latitude and longitude.

```APIDOC
## GET /people.geo

### Description
Get legislators for a given location.

### Method
GET

### Endpoint
https://v3.openstates.org/people.geo

### Parameters
#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)
- **lat** (number) - Required - Latitude of the location
- **lon** (number) - Required - Longitude of the location

### Request Example
GET https://v3.openstates.org/people.geo?lat=38.9072&lon=-77.0369&apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **results** (array) - List of legislators for the location

#### Response Example
{
  "results": [
    {
      "id": "ocd-person/1234",
      "name": "Jane Doe"
    }
  ]
}
```

--------------------------------

### GET /events/{event_id}

Source: https://docs.openstates.org/api-v3

Retrieves detailed information about a specific event using its internal ID.

```APIDOC
## GET /events/{event_id}

### Description
Get details on event by internal ID.

### Method
GET

### Endpoint
https://v3.openstates.org/events/{event_id}

### Parameters
#### Path Parameters
- **event_id** (string) - Required - The internal event identifier

#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)

### Request Example
GET https://v3.openstates.org/events/ocd-event/1234?apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **id** (string) - Event ID
- **name** (string) - Event name
- **jurisdiction** (string) - Jurisdiction ID

#### Response Example
{
  "id": "ocd-event/1234",
  "name": "Committee Hearing",
  "jurisdiction": "ocd-jurisdiction/country:us/state:ca/government"
}
```

--------------------------------

### GET /committees/{committee_id}

Source: https://docs.openstates.org/api-v3

Retrieves detailed information about a specific committee using its internal ID.

```APIDOC
## GET /committees/{committee_id}

### Description
Get details on committee by internal ID.

### Method
GET

### Endpoint
https://v3.openstates.org/committees/{committee_id}

### Parameters
#### Path Parameters
- **committee_id** (string) - Required - The internal committee identifier

#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)

### Request Example
GET https://v3.openstates.org/committees/ocd-committee/1234?apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **id** (string) - Committee ID
- **name** (string) - Committee name
- **jurisdiction** (string) - Jurisdiction ID

#### Response Example
{
  "id": "ocd-committee/1234",
  "name": "Committee on Finance",
  "jurisdiction": "ocd-jurisdiction/country:us/state:ca/government"
}
```

--------------------------------

### Show tables in DuckDB database

Source: https://docs.openstates.org/data/query-scraper-output-data

Use SHOW TABLES; to list the views and tables in your DuckDB database after creating the views.

```sql
SHOW TABLES;
```

--------------------------------

### GET /bills

Source: https://docs.openstates.org/api-v3

Searches for bills using various criteria such as jurisdiction, session, and bill ID.

```APIDOC
## GET /bills

### Description
Search bills by various criteria.

### Method
GET

### Endpoint
https://v3.openstates.org/bills

### Parameters
#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)
- **jurisdiction** (string) - Optional - Filter by jurisdiction
- **session** (string) - Optional - Filter by legislative session
- **bill_id** (string) - Optional - Filter by bill identifier

### Request Example
GET https://v3.openstates.org/bills?jurisdiction=ocd-jurisdiction/country:us/state:ca/government&session=2021&apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **results** (array) - List of bills

#### Response Example
{
  "results": [
    {
      "id": "ocd-bill/1234",
      "bill_id": "AB 100",
      "title": "An act to..."
    }
  ]
}
```

--------------------------------

### Show os-committees command help

Source: https://docs.openstates.org/contributing/people

Displays help for the os-committees command, which manages committee data.

```bash
poetry run os-committees --help

```

--------------------------------

### GET /jurisdictions/{jurisdiction_id}

Source: https://docs.openstates.org/api-v3

Retrieves detailed metadata for a specific jurisdiction identified by its jurisdiction ID.

```APIDOC
## GET /jurisdictions/{jurisdiction_id}

### Description
Get detailed metadata for a particular jurisdiction.

### Method
GET

### Endpoint
https://v3.openstates.org/jurisdictions/{jurisdiction_id}

### Parameters
#### Path Parameters
- **jurisdiction_id** (string) - Required - The jurisdiction identifier (e.g., ocd-jurisdiction/country:us/state:ca/government)

#### Query Parameters
- **apikey** (string) - Optional - API key (alternative to X-API-KEY header)

### Request Example
GET https://v3.openstates.org/jurisdictions/ocd-jurisdiction/country:us/state:ca/government?apikey=YOUR_API_KEY

### Response
#### Success Response (200)
- **id** (string) - Jurisdiction identifier
- **name** (string) - Jurisdiction name
- **classification** (string) - Jurisdiction classification

#### Response Example
{
  "id": "ocd-jurisdiction/country:us/state:ca/government",
  "name": "California",
  "classification": "government"
}
```
Tokens:
Raw