### Install HomeHarvest Library Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md Install the HomeHarvest library using pip. A Python version of 3.9 or higher is required. ```bash pip install -U homeharvest ``` -------------------------------- ### Scrape Property Listings with Sorting and Filtering Source: https://context7.com/zacharyhampton/homeharvest/llms.txt Examples of using scrape_property with various parameters to sort and filter real estate data. ```python # Newest listings first newest = scrape_property( location="Dallas, TX", listing_type="for_sale", sort_by="list_date", sort_direction="desc", past_days=7 ) ``` ```python # Cheapest properties first cheapest = scrape_property( location="Phoenix, AZ", listing_type="for_sale", sort_by="list_price", sort_direction="asc", price_max=300000 ) ``` ```python # Recently sold, highest price first top_sales = scrape_property( location="Beverly Hills, CA", listing_type="sold", sort_by="sold_date", sort_direction="desc", past_days=30 ) ``` ```python # Largest homes by square footage largest = scrape_property( location="Scottsdale, AZ", listing_type="for_sale", sort_by="sqft", sort_direction="desc", sqft_min=3000 ) ``` ```python # Most recently updated (price changes, status updates) recently_updated = scrape_property( location="San Jose, CA", listing_type="for_sale", sort_by="last_update_date", sort_direction="desc", updated_in_past_hours=24 ) ``` -------------------------------- ### Manage Pagination and Large Datasets Source: https://context7.com/zacharyhampton/homeharvest/llms.txt Techniques for fetching large result sets using offsets, limits, and date-based chunking to bypass API constraints. ```python from homeharvest import scrape_property from datetime import datetime, timedelta import pandas as pd # Basic pagination - fetch in chunks first_batch = scrape_property( location="New York, NY", listing_type="for_sale", limit=200, offset=0 # Results 0-199 ) second_batch = scrape_property( location="New York, NY", listing_type="for_sale", limit=200, offset=200 # Results 200-399 ) # Combine batches all_properties = pd.concat([first_batch, second_batch], ignore_index=True) # For large datasets exceeding 10k limit, use date ranges def scrape_large_dataset(location, listing_type, start_date, end_date, chunk_days=30): """Fetch properties in date chunks to bypass 10k limit.""" all_data = [] current = start_date while current < end_date: chunk_end = min(current + timedelta(days=chunk_days), end_date) properties = scrape_property( location=location, listing_type=listing_type, date_from=current, date_to=chunk_end, limit=10000 ) if not properties.empty: all_data.append(properties) print(f"{current.date()} to {chunk_end.date()}: {len(properties)} properties") current = chunk_end + timedelta(days=1) return pd.concat(all_data, ignore_index=True) if all_data else pd.DataFrame() # Fetch all sold properties in Los Angeles for 2024 la_2024_sold = scrape_large_dataset( location="Los Angeles, CA", listing_type="sold", start_date=datetime(2024, 1, 1), end_date=datetime(2024, 12, 31), chunk_days=30 ) # Sequential mode for narrow time windows (better rate limiting) recent_updates = scrape_property( location="Houston, TX", listing_type="for_sale", updated_in_past_hours=6, parallel=False, # Fetch pages sequentially with early termination sort_by="last_update_date", sort_direction="desc" ) ``` -------------------------------- ### Pagination Control Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md Configure pagination behavior by setting 'parallel=False' for sequential fetching. This mode can terminate early when filters no longer match, improving efficiency for narrow searches. ```python # Sequential mode with early termination (more efficient for narrow filters) properties = scrape_property( location="Los Angeles, CA", listing_type="for_sale", updated_in_past_hours=2, # Narrow time window parallel=False # Fetch pages sequentially, stop when filters no longer match ) ``` -------------------------------- ### Retrieve Raw Property Data Source: https://context7.com/zacharyhampton/homeharvest/llms.txt Fetches property data as raw dictionaries instead of a DataFrame. ```python raw_properties = scrape_property( location="Chicago, IL", listing_type="sold", past_days=7, return_type="raw" ) for prop in raw_properties[:2]: print(prop.keys()) # dict_keys(['property_url', 'property_id', 'listing_id', 'mls', 'status', # 'address', 'list_price', 'description', 'advertisers', ...]) ``` -------------------------------- ### Configure return types for property data Source: https://context7.com/zacharyhampton/homeharvest/llms.txt HomeHarvest supports returning data as Pandas DataFrames, Pydantic models, or raw JSON. Use the return_type parameter to specify the desired format. ```python from homeharvest import scrape_property # Default: Pandas DataFrame - great for analysis and export df_properties = scrape_property( location="Portland, OR", listing_type="for_sale", return_type="pandas" # Default ) print(df_properties.columns.tolist()) # ['property_url', 'property_id', 'listing_id', 'mls', 'mls_id', 'status', # 'mls_status', 'text', 'style', 'formatted_address', 'street', 'city', # 'state', 'zip_code', 'beds', 'full_baths', 'half_baths', 'sqft', ...] df_properties.to_excel("portland_listings.xlsx", index=False) # Pydantic models - type-safe access with full IDE support pydantic_properties = scrape_property( location="Boston, MA", listing_type="for_sale", return_type="pydantic" ) for prop in pydantic_properties[:3]: print(f"URL: {prop.property_url}") print(f"Address: {prop.address.formatted_address}") print(f"Price: ${prop.list_price:,}") if prop.description: print(f"Details: {prop.description.beds} bed, {prop.description.baths_full} bath") print(f"Size: {prop.description.sqft} sqft, built {prop.description.year_built}") if prop.advertisers and prop.advertisers.agent: print(f"Agent: {prop.advertisers.agent.name}") print("---") ``` -------------------------------- ### Filter Properties by Type and Status Source: https://context7.com/zacharyhampton/homeharvest/llms.txt Demonstrates filtering by specific property types and listing statuses, including calculating custom metrics on the returned data. ```python from homeharvest import scrape_property # Property types available: # 'single_family', 'multi_family', 'condos', 'condo_townhome_rowhome_coop', # 'condo_townhome', 'townhomes', 'duplex_triplex', 'farm', 'land', 'mobile' # Search for land listings land_properties = scrape_property( location="78028", # Kerrville, TX zip code listing_type="sold", property_type=["land"], past_days=365 ) # Calculate price per acre import pandas as pd land_properties["lot_acres"] = land_properties["lot_sqft"] / 43560 land_properties["price_per_acre"] = land_properties["sold_price"] / land_properties["lot_acres"] print(land_properties[["formatted_address", "sold_price", "lot_acres", "price_per_acre"]]) # Listing types available: # 'for_sale', 'for_rent', 'sold', 'pending', 'off_market', # 'new_community', 'other', 'ready_to_build' # Get all rental properties (apartments, condos, houses) rentals = scrape_property( location="San Francisco, CA", listing_type="for_rent", property_type=["apartment", "condos", "single_family"], price_max=4000 # Monthly rent limit ) # Search foreclosures only foreclosures = scrape_property( location="Phoenix, AZ", listing_type="for_sale", foreclosure=True, mls_only=True # Only MLS-listed properties ) # Get pending sales (under contract) pending = scrape_property( location="Nashville, TN", listing_type="pending", past_days=30 ) # Search all common listing types at once (returns for_sale, for_rent, sold, pending, off_market) all_listings = scrape_property( location="Atlanta, GA", listing_type=None # None returns all common types ) ``` -------------------------------- ### Sort Property Results Source: https://context7.com/zacharyhampton/homeharvest/llms.txt Defines available fields and directions for sorting search results. ```python from homeharvest import scrape_property # Sort options: 'list_date', 'sold_date', 'list_price', 'sqft', 'beds', 'baths', 'last_update_date' # Direction: 'asc' (ascending) or 'desc' (descending) ``` -------------------------------- ### Using Pydantic Models for Output Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md Retrieve scraped properties as Pydantic models for type safety and data validation. This allows for structured access to property details like address and description fields. ```python from homeharvest import scrape_property # Get properties as Pydantic models for type safety and data validation properties = scrape_property( location="San Diego, CA", listing_type="for_sale", return_type="pydantic" # Returns list of Property models ) # Access model fields with full type hints and validation for prop in properties[:5]: print(f"Address: {prop.address.formatted_address}") print(f"Price: ${prop.list_price:,}") if prop.description: print(f"Beds: {prop.description.beds}, Baths: {prop.description.baths_full}") ``` -------------------------------- ### Access Property Data via Pydantic Models Source: https://context7.com/zacharyhampton/homeharvest/llms.txt Retrieve fully typed Property objects by setting return_type to 'pydantic'. This allows for structured access to address, description, pricing, and agent information. ```python from homeharvest import scrape_property from homeharvest.core.scrapers.models import Property, Address, Description, Advertisers # Get properties as Pydantic models properties = scrape_property( location="Malibu, CA", listing_type="for_sale", return_type="pydantic", limit=10 ) for prop in properties: # Basic property info print(f"ID: {prop.property_id}") print(f"URL: {prop.property_url}") print(f"Status: {prop.status}") print(f"MLS: {prop.mls} #{prop.mls_id}") # Address details if prop.address: addr = prop.address print(f"Address: {addr.formatted_address}") print(f"Street: {addr.street}, Unit: {addr.unit}") print(f"City: {addr.city}, State: {addr.state}, ZIP: {addr.zip}") # Property description if prop.description: desc = prop.description print(f"Style: {desc.style}") print(f"Beds: {desc.beds}, Full Baths: {desc.baths_full}, Half Baths: {desc.baths_half}") print(f"Sqft: {desc.sqft}, Lot Sqft: {desc.lot_sqft}") print(f"Year Built: {desc.year_built}, Stories: {desc.stories}") print(f"Garage: {desc.garage}") # Pricing print(f"List Price: ${prop.list_price:,}" if prop.list_price else "List Price: N/A") print(f"Price/Sqft: ${prop.prc_sqft}" if prop.prc_sqft else "") print(f"Days on MLS: {prop.days_on_mls}") print(f"HOA Fee: ${prop.hoa_fee}/month" if prop.hoa_fee else "No HOA") # Dates print(f"List Date: {prop.list_date}") print(f"Last Update: {prop.last_update_date}") print(f"Pending Date: {prop.pending_date}" if prop.pending_date else "") print(f"Last Sold: {prop.last_sold_date} for ${prop.last_sold_price:,}" if prop.last_sold_date else "") # Location print(f"Coordinates: ({prop.latitude}, {prop.longitude})") print(f"County: {prop.county}, FIPS: {prop.fips_code}") print(f"Neighborhood: {prop.neighborhoods}") # Agent/Broker info if prop.advertisers: if prop.advertisers.agent: agent = prop.advertisers.agent print(f"Agent: {agent.name}, Email: {agent.email}") print(f"License: {agent.state_license}") if prop.advertisers.broker: print(f"Broker: {prop.advertisers.broker.name}") if prop.advertisers.office: print(f"Office: {prop.advertisers.office.name}") # Tax and valuation print(f"Assessed Value: ${prop.assessed_value:,}" if prop.assessed_value else "") print(f"Estimated Value: ${prop.estimated_value:,}" if prop.estimated_value else "") # Property flags if prop.flags: flags = [] if prop.flags.is_new_construction: flags.append("New Construction") if prop.flags.is_foreclosure: flags.append("Foreclosure") if prop.flags.is_pending: flags.append("Pending") if prop.flags.is_price_reduced: flags.append("Price Reduced") print(f"Flags: {', '.join(flags)}" if flags else "") print("=" * 50) ``` -------------------------------- ### Comprehensive Exception Handling in Python Source: https://context7.com/zacharyhampton/homeharvest/llms.txt Use this wrapper function to catch and handle specific HomeHarvest exceptions like `InvalidListingType`, `InvalidDate`, `AuthenticationError`, and general `ValueError` for robust data scraping. ```python from homeharvest import scrape_property from homeharvest.exceptions import InvalidListingType, InvalidDate, AuthenticationError def safe_scrape(location, listing_type, **kwargs): """Wrapper with comprehensive error handling.""" try: properties = scrape_property( location=location, listing_type=listing_type, **kwargs ) return properties except InvalidListingType as e: # Invalid listing_type value provided # Valid: 'for_sale', 'for_rent', 'sold', 'pending', 'off_market', # 'new_community', 'other', 'ready_to_build' print(f"Invalid listing type: {e}") return None except InvalidDate as e: # Date format error or date_to before date_from print(f"Date validation error: {e}") return None except AuthenticationError as e: # Realtor.com token request failed (rate limited or blocked) print(f"Authentication failed: {e}") print("Try using a proxy or wait before retrying") return None except ValueError as e: # Parameter validation errors (limit, offset, filters, sort) print(f"Validation error: {e}") return None # Example usage with error handling properties = safe_scrape( location="New York, NY", listing_type="for_sale", beds_min=2, price_max=500000 ) if properties is not None and not properties.empty: print(f"Found {len(properties)} properties") ``` ```python # Using proxy for rate limit issues properties = scrape_property( location="San Francisco, CA", listing_type="for_sale", proxy="http://user:pass@proxy.example.com:8080" ) ``` -------------------------------- ### Sorting and Listing Types Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md Customize sorting of results by fields like 'list_price' or 'list_date' and specify the direction ('asc' or 'desc'). Multiple listing types can be queried simultaneously using a list. ```python # Sort options: list_price, list_date, sqft, beds, baths, last_update_date # Listing types: "for_sale", "for_rent", "sold", "pending", "off_market", list, or None (common types) properties = scrape_property( location="Miami, FL", listing_type=["for_sale", "pending"], # Single string, list, or None sort_by="list_price", # Sort field sort_direction="asc", # "asc" or "desc" limit=100 ) ``` -------------------------------- ### Basic Property Scraping Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md Scrape properties for a given location and listing type, filtering by days. Results are returned as a Pandas DataFrame and can be saved to CSV. ```python from homeharvest import scrape_property properties = scrape_property( location="San Diego, CA", listing_type="sold", # for_sale, for_rent, pending past_days=30 ) properties.to_csv("results.csv", index=False) print(f"Found {len(properties)} properties") ``` -------------------------------- ### Property Attribute Filtering Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md Apply detailed filters for property attributes such as minimum and maximum beds, baths, square footage, price, lot size, and year built. ```python # Combine any filters: beds, baths, sqft, price, lot_sqft, year_built properties = scrape_property( location="San Francisco, CA", listing_type="for_sale", beds_min=3, beds_max=5, baths_min=2.0, sqft_min=1500, sqft_max=3000, price_min=300000, price_max=800000, year_built_min=2000, lot_sqft_min=5000 ) ``` -------------------------------- ### Scrape property data with HomeHarvest Source: https://context7.com/zacharyhampton/homeharvest/llms.txt The scrape_property function is the primary entry point for fetching real estate listings. It supports various filters like location, listing type, property attributes, and time-based queries. ```python from homeharvest import scrape_property from datetime import datetime, timedelta # Basic search - get sold properties in San Diego from the last 30 days properties = scrape_property( location="San Diego, CA", listing_type="sold", past_days=30 ) print(f"Found {len(properties)} properties") properties.to_csv("san_diego_sold.csv", index=False) # Advanced search with multiple filters properties = scrape_property( location="Austin, TX", listing_type="for_sale", property_type=["single_family", "townhomes"], beds_min=3, beds_max=5, baths_min=2.0, price_min=300000, price_max=600000, sqft_min=1500, year_built_min=2010, sort_by="list_price", sort_direction="asc", limit=500 ) # Search multiple listing types at once properties = scrape_property( location="Miami, FL", listing_type=["for_sale", "pending"], # Returns properties matching ANY status past_days=14 ) # Time-based filtering with hours precision properties = scrape_property( location="Los Angeles, CA", listing_type="for_sale", past_hours=24, # Properties listed in last 24 hours parallel=False # Sequential mode for efficient early termination ) # Date range search with datetime objects properties = scrape_property( location="Denver, CO", listing_type="sold", date_from=datetime(2024, 1, 1), date_to=datetime(2024, 6, 30) ) # Filter by last update date properties = scrape_property( location="Seattle, WA", listing_type="for_sale", updated_in_past_hours=48 # Recently updated listings ) # Search by address with radius properties = scrape_property( location="1600 Pennsylvania Ave, Washington DC", listing_type="for_sale", radius=2.0 # Miles from address ) ``` -------------------------------- ### Scrape Property Data Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md The `scrape_property` function enables fetching real estate listings based on a wide range of criteria. It supports flexible location inputs, various listing types, and detailed filtering options. ```APIDOC ## POST /api/properties/scrape ### Description Fetches real estate property listings based on specified criteria. ### Method POST ### Endpoint /api/properties/scrape ### Parameters #### Request Body - **location** (str) - Required - Flexible location search. Accepts ZIP code, City, City, State, Full address, Neighborhood, County, or State. - **listing_type** (str | list[str] | None) - Optional - Type of listing. Options: 'for_sale', 'for_rent', 'sold', 'pending', 'off_market', 'new_community', 'other', 'ready_to_build'. Can be a list for OR conditions. Defaults to common types if None. - **property_type** (list) - Optional - Type of properties to filter by. Examples: 'single_family', 'multi_family', 'condos', 'townhomes', 'farm', 'land', 'mobile'. - **return_type** (str) - Optional - Format of the returned data. Options: 'pandas' (default), 'pydantic', 'raw'. - **radius** (decimal) - Optional - Radius in miles for searches based on a specific address. - **past_days** (integer) - Optional - Number of past days to filter properties (e.g., for 'sold' or 'list_date'). - **past_hours** (integer | timedelta) - Optional - Number of past hours to filter properties (client-side filtering). Cannot be used with `past_days` or `date_from`/`date_to`. - **date_from** (string) - Optional - Start date for filtering (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS). Required if `date_to` is provided. - **date_to** (string) - Optional - End date for filtering (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS). Required if `date_from` is provided. - **updated_since** (datetime | str) - Optional - Filter properties updated since a specific date/time. - **updated_in_past_hours** (integer | timedelta) - Optional - Filter properties updated in the past X hours. - **beds_min** (integer) - Optional - Minimum number of bedrooms. - **beds_max** (integer) - Optional - Maximum number of bedrooms. - **baths_min** (float) - Optional - Minimum number of bathrooms. - **baths_max** (float) - Optional - Maximum number of bathrooms. - **sqft_min** (integer) - Optional - Minimum square footage. - **sqft_max** (integer) - Optional - Maximum square footage. - **price_min** (integer) - Optional - Minimum listing price. - **price_max** (integer) - Optional - Maximum listing price. - **lot_sqft_min** (integer) - Optional - Minimum lot size in square feet. - **lot_sqft_max** (integer) - Optional - Maximum lot size in square feet. - **year_built_min** (integer) - Optional - Minimum year built. - **year_built_max** (integer) - Optional - Maximum year built. - **sort_by** (string) - Optional - Field to sort results by. Options: 'list_date', 'sold_date', 'list_price', 'sqft', 'beds', 'baths', 'last_update_date'. - **sort_direction** (string) - Optional - Direction of sorting. Options: 'asc', 'desc' (default). - **mls_only** (boolean) - Optional - If True, fetches only MLS listings. - **foreclosure** (boolean) - Optional - If True, fetches only foreclosures. - **proxy** (string) - Optional - Proxy server to use for the request (format: 'http://user:pass@host:port'). - **extra_property_data** (boolean) - Optional - If True, fetches additional property data (e.g., schools, tax appraisals). - **exclude_pending** (boolean) - Optional - If True, excludes 'pending' properties from 'for_sale' results. - **limit** (integer) - Optional - Maximum number of properties to fetch (max & default: 10000). ### Request Example ```json { "location": "San Diego, CA", "listing_type": "for_sale", "property_type": ["single_family", "condos"], "beds_min": 3, "price_max": 750000, "sort_by": "list_price", "sort_direction": "asc" } ``` ### Response #### Success Response (200) - **properties** (list) - A list of property objects matching the criteria. - **return_type** (str) - The format of the returned data ('pandas', 'pydantic', or 'raw'). #### Response Example (raw JSON) ```json { "properties": [ { "id": "12345", "address": "123 Main St, San Diego, CA 92101", "price": 650000, "beds": 3, "baths": 2.5, "sqft": 1800, "listing_type": "for_sale" } // ... more properties ], "return_type": "raw" } ``` ``` -------------------------------- ### Flexible Location Formats Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md The scrape_property function accepts various location formats, including zip codes, cities, 'city, state', or full addresses. An optional radius can be specified to search within a certain distance. ```python # Accepts: zip code, city, "city, state", full address, etc. properties = scrape_property( location="San Diego, CA", # or "92104", "San Diego", "1234 Main St, San Diego, CA 92104" radius=5.0 # Optional: search within radius (miles) of address ) ``` -------------------------------- ### Time-Based Filtering Source: https://github.com/zacharyhampton/homeharvest/blob/master/README.md Filter properties based on time using 'past_hours' or 'past_days', or by providing datetime objects for 'date_from' and 'date_to'. The library automatically detects hour precision. ```python from datetime import datetime, timedelta # Filter by hours or use datetime/timedelta objects properties = scrape_property( location="Austin, TX", listing_type="for_sale", past_hours=24, # or timedelta(hours=24) for Pythonic approach # date_from=datetime.now() - timedelta(days=7), # Alternative: datetime objects # date_to=datetime.now(), # Automatic hour precision detection ) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.