### Install JobSpy Python Package Source: https://github.com/speedyapply/jobspy/blob/main/README.md Install the JobSpy library using pip. Ensure you are using Python version 3.10 or higher. ```bash pip install -U python-jobspy ``` -------------------------------- ### Build Indeed Search Query Source: https://github.com/speedyapply/jobspy/blob/main/README.md Construct a precise search query for Indeed to filter job roles effectively. Use hyphens to exclude terms and quotes for exact matches. ```python search_term='"engineering intern" software summer (java OR python OR c++) 2025 -tax -marketing' ``` -------------------------------- ### Proxy Support — Bypass rate limiting with rotating proxies Source: https://context7.com/speedyapply/jobspy/llms.txt This section details how to use the `proxies` and `ca_cert` parameters within `scrape_jobs` to manage rate limiting by rotating through a list of proxy addresses. ```APIDOC ## Proxy Support — Bypass rate limiting with rotating proxies ### Description Every scraper accepts a `proxies` parameter that round-robins requests across a list of proxy addresses. Supports plain `host:port`, `user:pass@host:port`, and `localhost` (to bypass proxy for that turn). An optional CA certificate path can be provided for proxy SSL validation. ### Method ```python scrape_jobs( site_name: Optional[Union[str, List[str]]] = None, search_term: Optional[str] = None, location: Optional[str] = None, results_wanted: int = 100, proxies: Optional[List[str]] = None, ca_cert: Optional[str] = None, hours_old: Optional[int] = None, # ... other parameters ) ``` ### Parameters - **proxies** (List[str], optional): A list of proxy addresses in the format `host:port`, `user:pass@host:port`, or `localhost`. Requests will be rotated through these proxies. - **ca_cert** (str, optional): Path to a CA certificate file for SSL validation of proxies. ### Request Example ```python from jobspy import scrape_jobs jobs = scrape_jobs( site_name=["linkedin", "indeed"], search_term="data scientist", location="New York, NY", results_wanted=50, proxies=[ "user:password@192.168.1.1:8080", "208.195.175.46:65095", "localhost", ], ca_cert="/path/to/ca-bundle.crt", hours_old=72, ) print(f"Scraped {len(jobs)} jobs using rotating proxies") print(jobs[["site", "title", "company"]].head()) ``` ### Response #### Success Response (pandas DataFrame) Returns a pandas DataFrame containing the scraped job postings. The structure is the same as the main `scrape_jobs` function response. #### Response Example ```python print(jobs[["site", "title", "company"]].head()) ``` ``` -------------------------------- ### Google Jobs Search with Natural Language Queries Source: https://context7.com/speedyapply/jobspy/llms.txt For Google Jobs, use google_search_term to mirror Google's job search UI queries. The regular search_term and location parameters are ignored. Tips include using 'near [city, state]' and recency terms like 'since last week'. ```python from jobspy import scrape_jobs jobs = scrape_jobs( site_name="google", google_search_term="software engineer jobs near Austin, TX since last week", results_wanted=20, verbose=1, ) print(f"Google jobs found: {len(jobs)}") print(jobs[["title", "company", "location", "date_posted", "job_url"]].head(10)) ``` -------------------------------- ### Scrape Jobs from Multiple Sites Source: https://github.com/speedyapply/jobspy/blob/main/README.md Scrape job postings from specified sites using keywords and location. The results are returned in a pandas DataFrame. You can customize search terms, location, and other parameters. ```python import csv from jobspy import scrape_jobs jobs = scrape_jobs( site_name=["indeed", "linkedin", "zip_recruiter", "google"], # "glassdoor", "bayt", "naukri", "bdjobs" search_term="software engineer", google_search_term="software engineer jobs near San Francisco, CA since yesterday", location="San Francisco, CA", results_wanted=20, hours_old=72, country_indeed='USA', # linkedin_fetch_description=True # gets more info such as description, direct job url (slower) # proxies=["208.195.175.46:65095", "208.195.175.45:65095", "localhost"], ) print(f"Found {len(jobs)} jobs") print(jobs.head()) jobs.to_csv("jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\", index=False) # to_excel ``` -------------------------------- ### Pagination with `offset` Source: https://context7.com/speedyapply/jobspy/llms.txt Use the `offset` parameter to skip results, enabling pagination or resuming interrupted scrapes. This allows stepping through large result sets when combined with `results_wanted`. ```python from jobspy import scrape_jobs import pandas as pd all_jobs = [] # Collect 3 pages of 25 results each from Indeed for page in range(3): batch = scrape_jobs( site_name="indeed", search_term="devops engineer", location="Remote", country_indeed="USA", results_wanted=25, offset=page * 25, hours_old=168, verbose=0, ) all_jobs.append(batch) print(f"Page {page + 1}: {len(batch)} jobs") combined = pd.concat(all_jobs, ignore_index=True).drop_duplicates(subset=["job_url"]) print(f"Total unique jobs: {len(combined)}") combined.to_csv("devops_jobs_all_pages.csv", index=False) ``` -------------------------------- ### JobPost Model Schema Overview Source: https://context7.com/speedyapply/jobspy/llms.txt Each row in the returned DataFrame represents a `JobPost` Pydantic model. Understanding this schema is crucial for downstream filtering, storage, and analysis of job postings. ```python from jobspy import scrape_jobs jobs = scrape_jobs( site_name=["indeed", "naukri"], search_term="data analyst", location="Mumbai", country_indeed="india", results_wanted=10, verbose=0, ) # Core fields (all boards) print(jobs[["id", "site", "title", "company", "location", "date_posted", "is_remote"]].head()) # Compensation fields print(jobs[["interval", "min_amount", "max_amount", "currency", "salary_source"]].head()) # Indeed-specific company metadata indeed_jobs = jobs[jobs["site"] == "indeed"] print(indeed_jobs[["company_addresses", "company_num_employees", "company_revenue", "company_description"]].head()) # Naukri-specific fields naukri_jobs = jobs[jobs["site"] == "naukri"] print(naukri_jobs[["skills", "experience_range", "company_rating", "company_reviews_count", "vacancy_count", "work_from_home_type"]].head()) ``` -------------------------------- ### Normalize Compensation to Yearly Figures with JobSpy Source: https://context7.com/speedyapply/jobspy/llms.txt When enforce_annual_salary is True, JobSpy converts hourly, daily, weekly, and monthly compensation values to annual equivalents. This ensures all salary figures are annual regardless of the original interval. ```python from jobspy import scrape_jobs jobs = scrape_jobs( site_name=["indeed", "zip_recruiter"], search_term="nurse", location="Dallas, TX", country_indeed="USA", results_wanted=20, enforce_annual_salary=True, # all min_amount/max_amount become yearly verbose=0, ) # All salary figures are now annual regardless of original interval salary_jobs = jobs[jobs["min_amount"].notna()] print(salary_jobs[["title", "company", "interval", "min_amount", "max_amount"]].head()) # interval will be "yearly" for all converted rows ``` -------------------------------- ### Multi-Country Search on Indeed and Glassdoor Source: https://context7.com/speedyapply/jobspy/llms.txt Search jobs in 60+ countries using the `country_indeed` parameter for Indeed and Glassdoor. LinkedIn searches globally with `location` only. ZipRecruiter is limited to US/Canada. ```python from jobspy import scrape_jobs # Search for jobs in Germany on Indeed and Glassdoor jobs = scrape_jobs( site_name=["indeed", "glassdoor"], search_term="backend developer", location="Berlin", country_indeed="germany", # use exact country name (see supported list) results_wanted=20, description_format="html", # "markdown", "html", or "plain" verbose=2, ) print(jobs[["site", "title", "company", "location", "currency"]].head(10)) # Supported country examples: "australia", "canada", "france", "germany", # "india", "japan", "netherlands", "singapore", "uk", "usa" # Full list in README — countries marked * also support Glassdoor ``` -------------------------------- ### Scrape Jobs with Proxy Support Source: https://context7.com/speedyapply/jobspy/llms.txt Scrapes jobs using a list of rotating proxies to bypass rate limiting. Supports various proxy formats and an optional CA certificate for SSL validation. ```python from jobspy import scrape_jobs jobs = scrape_jobs( site_name=["linkedin", "indeed"], search_term="data scientist", location="New York, NY", results_wanted=50, proxies=[ "user:password@192.168.1.1:8080", "208.195.175.46:65095", "localhost", # falls back to direct connection in rotation ], ca_cert="/path/to/ca-bundle.crt", # optional, for proxy SSL hours_old=72, ) print(f"Scraped {len(jobs)} jobs using rotating proxies") print(jobs[["site", "title", "company"]].head()) ``` -------------------------------- ### Scrape Jobs from Multiple Boards Source: https://context7.com/speedyapply/jobspy/llms.txt Scrapes jobs from specified boards concurrently for a given search term and location. Supports filtering by hours old, job type, remote status, distance, and salary normalization. `verbose` controls logging level. ```python import csv from jobspy import scrape_jobs # Scrape multiple job boards concurrently for a software engineering role jobs = scrape_jobs( site_name=["indeed", "linkedin", "zip_recruiter", "glassdoor", "google"], search_term="software engineer", google_search_term="software engineer jobs near San Francisco, CA since yesterday", location="San Francisco, CA", results_wanted=25, # per site hours_old=48, # only jobs posted in the last 48 hours job_type="fulltime", is_remote=False, distance=25, # miles country_indeed="USA", enforce_annual_salary=True, # normalize hourly/monthly to yearly description_format="markdown", verbose=1, # 0=errors only, 1=warnings+errors, 2=all ) print(f"Found {len(jobs)} jobs") print(jobs[["site", "title", "company", "location", "min_amount", "max_amount", "currency"]].head(10)) # Export to CSV jobs.to_csv("sf_software_jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\\", index=False) # Export to Excel jobs.to_excel("sf_software_jobs.xlsx", index=False) # Expected output columns: # id, site, job_url, job_url_direct, title, company, location, date_posted, # job_type, salary_source, interval, min_amount, max_amount, currency, # is_remote, job_level, job_function, listing_type, emails, description, # company_industry, company_url, company_logo, company_url_direct, # company_addresses, company_num_employees, company_revenue, company_description ``` -------------------------------- ### LinkedIn-Specific Parameters — Full descriptions and company filtering Source: https://context7.com/speedyapply/jobspy/llms.txt This section covers additional parameters available exclusively for LinkedIn scraping: `linkedin_fetch_description` to retrieve full job details and `linkedin_company_ids` to filter by specific companies. ```APIDOC ## LinkedIn-Specific Parameters — Full descriptions and company filtering ### Description LinkedIn scraping supports two extra parameters: `linkedin_fetch_description` (fetches the full job description and direct apply URL via an additional per-job HTTP request) and `linkedin_company_ids` (restricts results to specific company IDs). Note that enabling `linkedin_fetch_description` multiplies the number of requests by O(n). ### Method ```python scrape_jobs( site_name="linkedin", search_term: Optional[str] = None, location: Optional[str] = None, results_wanted: int = 100, linkedin_fetch_description: bool = False, linkedin_company_ids: Optional[List[int]] = None, job_type: Optional[str] = None, hours_old: Optional[int] = None, # ... other parameters ) ``` ### Parameters - **linkedin_fetch_description** (bool, optional): If True, fetches the full job description and direct apply URL for each listing. This significantly increases the number of requests and processing time. Defaults to False. - **linkedin_company_ids** (List[int], optional): A list of LinkedIn company IDs to filter the job search results. Only jobs from these companies will be returned. ### Request Example ```python from jobspy import scrape_jobs jobs = scrape_jobs( site_name="linkedin", search_term="machine learning engineer", location="Seattle, WA", results_wanted=15, linkedin_fetch_description=True, linkedin_company_ids=[1441, 10667], # 1441=Google, 10667=Amazon job_type="fulltime", hours_old=168, # last 7 days ) # LinkedIn-specific fields populated when linkedin_fetch_description=True: print(jobs[["title", "company", "job_level", "company_industry", "job_url_direct", "description"]].head()) ``` ### Response #### Success Response (pandas DataFrame) Returns a pandas DataFrame containing job postings. When `linkedin_fetch_description` is True, the following fields are populated: - **job_url_direct**: Direct URL to apply for the job. - **description**: Full job description. - **job_level**: The level of the job (e.g., Senior, Junior). - **company_industry**: The industry of the company. #### Response Example ```python print(jobs[["title", "company", "job_level", "company_industry", "job_url_direct", "description"]].head()) ``` ``` -------------------------------- ### Scrape LinkedIn Jobs with Full Descriptions and Company Filtering Source: https://context7.com/speedyapply/jobspy/llms.txt Scrapes LinkedIn jobs, optionally fetching full descriptions and direct apply URLs, and filtering by specific company IDs. Enabling full description fetching significantly increases request count and processing time. ```python from jobspy import scrape_jobs # Scrape LinkedIn-only with full descriptions, filtered to specific companies jobs = scrape_jobs( site_name="linkedin", search_term="machine learning engineer", location="Seattle, WA", results_wanted=15, linkedin_fetch_description=True, # slower, but gets full description + direct URL linkedin_company_ids=[1441, 10667], # 1441=Google, 10667=Amazon job_type="fulltime", hours_old=168, # last 7 days ) # LinkedIn-specific fields populated when linkedin_fetch_description=True: print(jobs[["title", "company", "job_level", "company_industry", "job_url_direct", "description"]].head()) ``` -------------------------------- ### `scrape_jobs()` — Scrape jobs from one or more job boards Source: https://context7.com/speedyapply/jobspy/llms.txt The primary function to scrape jobs from various job boards. It supports concurrent scraping, rich filtering parameters, and returns results as a normalized pandas DataFrame. All parameters are optional; omitting `site_name` scrapes all supported boards. ```APIDOC ## `scrape_jobs()` — Scrape jobs from one or more job boards ### Description The primary and only public function in JobSpy. It launches site-specific scrapers concurrently, merges all results into a single pandas DataFrame sorted by site and posting date, and applies salary normalization when requested. All parameters are optional; calling without `site_name` will scrape all supported boards. ### Method ```python scrape_jobs( site_name: Optional[Union[str, List[str]]] = None, search_term: Optional[str] = None, google_search_term: Optional[str] = None, location: Optional[str] = None, results_wanted: int = 100, hours_old: Optional[int] = None, job_type: Optional[str] = None, is_remote: Optional[bool] = None, distance: int = 25, country_indeed: Optional[str] = None, enforce_annual_salary: bool = False, description_format: Optional[str] = None, verbose: int = 0, proxies: Optional[List[str]] = None, ca_cert: Optional[str] = None, linkedin_fetch_description: bool = False, linkedin_company_ids: Optional[List[int]] = None, ) ``` ### Parameters #### Common Parameters - **site_name** (str or List[str], optional): The job board(s) to scrape. Defaults to all supported boards if not provided. - **search_term** (str, optional): The primary search query. - **google_search_term** (str, optional): A more specific search term for Google jobs. - **location** (str, optional): The desired job location. - **results_wanted** (int, optional): The number of results to retrieve per site. Defaults to 100. - **hours_old** (int, optional): Filter jobs posted within the last specified hours. - **job_type** (str, optional): Type of job (e.g., 'fulltime', 'parttime'). - **is_remote** (bool, optional): Whether to search for remote positions. - **distance** (int, optional): Search radius in miles. Defaults to 25. - **country_indeed** (str, optional): Specific country for Indeed searches. - **enforce_annual_salary** (bool, optional): Normalize salaries to annual. Defaults to False. - **description_format** (str, optional): Format for job descriptions (e.g., 'markdown'). - **verbose** (int, optional): Verbosity level for logging (0=errors, 1=warnings+errors, 2=all). Defaults to 0. #### Proxy Support Parameters - **proxies** (List[str], optional): A list of proxy addresses to use for rotating requests. - **ca_cert** (str, optional): Path to a CA certificate bundle for proxy SSL validation. #### LinkedIn-Specific Parameters - **linkedin_fetch_description** (bool, optional): Fetch full job descriptions and direct apply URLs. Defaults to False. Note: This increases request count significantly. - **linkedin_company_ids** (List[int], optional): Filter results to specific LinkedIn company IDs. ### Request Example ```python from jobspy import scrape_jobs jobs = scrape_jobs( site_name=["indeed", "linkedin", "zip_recruiter", "glassdoor", "google"], search_term="software engineer", google_search_term="software engineer jobs near San Francisco, CA since yesterday", location="San Francisco, CA", results_wanted=25, hours_old=48, job_type="fulltime", is_remote=False, distance=25, country_indeed="USA", enforce_annual_salary=True, description_format="markdown", verbose=1, ) print(f"Found {len(jobs)} jobs") print(jobs[["site", "title", "company", "location", "min_amount", "max_amount", "currency"]].head(10)) ``` ### Response #### Success Response (pandas DataFrame) Returns a pandas DataFrame containing job postings with the following columns: - **id**: Unique identifier - **site**: Job board name - **job_url**: URL to the job posting - **job_url_direct**: Direct URL to apply (if `linkedin_fetch_description` is True) - **title**: Job title - **company**: Company name - **location**: Job location - **date_posted**: Date the job was posted - **job_type**: Type of employment (e.g., 'fulltime') - **salary_source**: Source of salary information - **interval**: Salary interval (e.g., 'yearly', 'hourly') - **min_amount**: Minimum salary amount - **max_amount**: Maximum salary amount - **currency**: Salary currency - **is_remote**: Boolean indicating if the job is remote - **job_level**: Job level (e.g., 'Senior', 'Junior') - **job_function**: Functional area of the job - **listing_type**: Type of listing - **emails**: Email addresses found in the posting - **description**: Job description - **company_industry**: Industry of the company - **company_url**: Company website URL - **company_logo**: URL of the company logo - **company_url_direct**: Direct URL to the company profile - **company_addresses**: Company addresses - **company_num_employees**: Number of employees - **company_revenue**: Company revenue - **company_description**: Description of the company #### Response Example ```python # Example of printing the head of the DataFrame print(jobs[["site", "title", "company", "location", "min_amount", "max_amount", "currency"]].head(10)) # Example of exporting to CSV # jobs.to_csv("sf_software_jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\", index=False) # Example of exporting to Excel # jobs.to_excel("sf_software_jobs.xlsx", index=False) ``` ``` -------------------------------- ### JobPost Schema Definition Source: https://github.com/speedyapply/jobspy/blob/main/README.md Defines the structure of a JobPost object, detailing fields common across job boards and those specific to platforms like LinkedIn, Indeed, and Naukri. ```plaintext JobPost ├── title ├── company ├── company_url ├── job_url ├── location │ ├── country │ ├── city │ ├── state ├── is_remote ├── description ├── job_type: fulltime, parttime, internship, contract ├── job_function │ ├── interval: yearly, monthly, weekly, daily, hourly │ ├── min_amount │ ├── max_amount │ ├── currency │ └── salary_source: direct_data, description (parsed from posting) ├── date_posted └── emails Linkedin specific └── job_level Linkedin & Indeed specific └── company_industry Indeed specific ├── company_country ├── company_addresses ├── company_employees_label ├── company_revenue_label ├── company_description └── company_logo Naukri specific ├── skills ├── experience_range ├── company_rating ├── company_reviews_count ├── vacancy_count └── work_from_home_type ``` -------------------------------- ### Indeed Advanced Search Syntax Source: https://context7.com/speedyapply/jobspy/llms.txt Use boolean operators and exclusion terms for precise filtering on Indeed. Note that only one of `hours_old`, `(job_type + is_remote)`, or `easy_apply` can be used per Indeed search. ```python from jobspy import scrape_jobs # Advanced Indeed query: exact phrase + required keywords + exclusions jobs = scrape_jobs( site_name="indeed", search_term='"software engineer" python (django OR fastapi OR flask) -manager -senior', location="Austin, TX", country_indeed="USA", results_wanted=30, hours_old=24, description_format="markdown", enforce_annual_salary=True, verbose=0, ) # Filter results in pandas for roles with salary data paid_jobs = jobs[jobs["min_amount"].notna()].copy() print(f"Jobs with salary info: {len(paid_jobs)} / {len(jobs)}") print(paid_jobs[["title", "company", "interval", "min_amount", "max_amount", "salary_source"]].head(10)) # salary_source values: # "direct_data" - salary explicitly listed on the job board # "description" - salary extracted by regex from description text (US only) ``` -------------------------------- ### Filter for `easy_apply` jobs Source: https://context7.com/speedyapply/jobspy/llms.txt The `easy_apply` flag restricts results to jobs with one-click apply options directly on the job board. Note that LinkedIn's Easy Apply filter has limited reliability. ```python from jobspy import scrape_jobs # Find jobs with one-click apply on ZipRecruiter jobs = scrape_jobs( site_name=["zip_recruiter", "indeed"], search_term="product manager", location="Chicago, IL", results_wanted=20, easy_apply=True, job_type="fulltime", country_indeed="USA", ) print(f"Easy apply jobs: {len(jobs)}") print(jobs[["site", "title", "company", "job_url"]].head()) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.