### Social Media Scraping Example (Twitter) Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Example of how to use the Apify Client to run a Twitter scraper and process the results. This demonstrates calling a specific actor for scraping. ```APIDOC ## POST /v2/acts/{actorId}/runs ### Description This example demonstrates how to use the `apify_client` library in Python to run the 'mikolabs/x-scraper' actor to scrape tweets based on a search query. ### Method POST (Implicitly via `client.actor(...).call()`) ### Endpoint (Internal Apify API endpoint for running actors, e.g., `/v2/acts/mikolabs/x-scraper/runs`) ### Parameters #### Request Body (for the actor) - **searchTerms** (array of strings) - Required - The search queries to use for scraping tweets. - **maxTweets** (integer) - Optional - The maximum number of tweets to scrape. Defaults to 100. - **includeReplies** (boolean) - Optional - Whether to include replies in the scraped results. Defaults to `false`. - **onlyVerified** (boolean) - Optional - Whether to only scrape tweets from verified accounts. Defaults to `false`. - **proxy** (object) - Optional - Proxy configuration. - **useApifyProxy** (boolean) - Required if `proxy` is used - Whether to use Apify's proxy service. Set to `true`. ### Request Example (Python Client) ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") run_input = { "searchTerms": ["artificial intelligence"], "maxTweets": 50, "includeReplies": False, "onlyVerified": False, "proxy": { "useApifyProxy": True } } # Run the Twitter scraper run = client.actor("mikolabs/x-scraper").call(run_input=run_input) ``` ### Response #### Success Response (from `client.actor(...).call()`) - **run** (object) - Information about the executed actor run. - **defaultDatasetId** (string) - The ID of the dataset where the scraped items are stored. - **status** (string) - The status of the run (e.g., 'SUCCEEDED'). #### Response Example (from `client.actor(...).call()`) ```json { "id": "some_run_id", "defaultDatasetId": "some_dataset_id", "status": "SUCCEEDED" } ``` #### Scraped Tweet Object Example (from dataset) ```json { "username": "example_user", "text": "This is a sample tweet content...", "likes": 120, "retweets": 30, "createdAt": "2023-10-27T10:00:00Z" } ``` ``` -------------------------------- ### Execute Apify API via REST Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Demonstrates how to run an Amazon Product Scraper API using a cURL command. It specifies the API endpoint, authentication token, and input parameters like start URLs and maximum items. The response includes a run ID for status tracking. ```bash # Run an Amazon Product Scraper API curl -X POST "https://api.apify.com/v2/acts/api-empire~amazon-product-scraper/runs?token=YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "startUrls": [ { "url": "https://www.amazon.com/dp/B09V3KXJPB" } ], "maxItems": 100 }' # Response includes run ID for checking status # { # "data": { # "id": "abc123xyz", # "actId": "api-empire~amazon-product-scraper", # "status": "RUNNING", # ... # } # } ``` -------------------------------- ### POST /v2/acts/{actorId}/runs Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Execute any scraping API by calling the Apify Actor Run endpoint with your API token and input parameters. This example shows how to run an Amazon Product Scraper API. ```APIDOC ## POST /v2/acts/{actorId}/runs ### Description Executes a specified Apify Actor (API) with provided input parameters. ### Method POST ### Endpoint `https://api.apify.com/v2/acts/{actorId}/runs?token=YOUR_APIFY_TOKEN` ### Parameters #### Path Parameters - **actorId** (string) - Required - The ID of the actor to run (e.g., `api-empire~amazon-product-scraper`). #### Query Parameters - **token** (string) - Required - Your Apify API token. #### Request Body - **startUrls** (array of objects) - Optional - An array of objects, each containing a `url` to start scraping from. - **url** (string) - Required - The URL to scrape. - **maxItems** (integer) - Optional - The maximum number of items to scrape. ### Request Example ```bash curl -X POST "https://api.apify.com/v2/acts/api-empire~amazon-product-scraper/runs?token=YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "startUrls": [ { "url": "https://www.amazon.com/dp/B09V3KXJPB" } ], "maxItems": 100 }' ``` ### Response #### Success Response (200) - **data** (object) - Contains information about the actor run. - **id** (string) - The unique ID of the actor run. - **actId** (string) - The ID of the actor that was run. - **status** (string) - The current status of the run (e.g., `RUNNING`). #### Response Example ```json { "data": { "id": "abc123xyz", "actId": "api-empire~amazon-product-scraper", "status": "RUNNING" } } ``` ``` -------------------------------- ### POST /v2/schedules Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Create scheduled tasks to run APIs at specified intervals using the Apify scheduling API. This example creates a scheduled task to run daily at 9 AM UTC. ```APIDOC ## POST /v2/schedules ### Description Creates a new scheduled task to automate the execution of Apify Actors. ### Method POST ### Endpoint `https://api.apify.com/v2/schedules?token=YOUR_APIFY_TOKEN` ### Parameters #### Query Parameters - **token** (string) - Required - Your Apify API token. #### Request Body - **name** (string) - Required - A descriptive name for the schedule. - **cronExpression** (string) - Required - A cron expression defining the schedule's timing (e.g., `0 9 * * *` for 9 AM UTC daily). - **isEnabled** (boolean) - Required - Whether the schedule is currently active. - **isExclusive** (boolean) - Required - If true, prevents overlapping runs of the same schedule. - **actions** (array of objects) - Required - A list of actions to perform when the schedule triggers. - **type** (string) - Required - The type of action, e.g., `RUN_ACTOR`. - **actorId** (string) - Required - The ID of the actor to run. - **runInput** (object) - Optional - The input parameters for the actor run. - **searchQuery** (string) - Example input for a search scraper. - **maxItems** (integer) - Example input for a search scraper. - **sortBy** (string) - Example input for a search scraper. ### Request Example ```bash curl -X POST "https://api.apify.com/v2/schedules?token=YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "daily-price-monitor", "cronExpression": "0 9 * * *", "isEnabled": true, "isExclusive": true, "actions": [{ "type": "RUN_ACTOR", "actorId": "scraper-engine~amazon-search-scraper", "runInput": { "searchQuery": "laptop", "maxItems": 50, "sortBy": "price-asc" } }] }' ``` ### Response #### Success Response (200) - **data** (object) - Contains information about the created schedule. - **id** (string) - The unique ID of the schedule. - **name** (string) - The name of the schedule. - **cronExpression** (string) - The cron expression for the schedule. - **nextRunAt** (string) - The timestamp of the next scheduled run. #### Response Example ```json { "data": { "id": "schedule123", "name": "daily-price-monitor", "cronExpression": "0 9 * * *", "nextRunAt": "2025-01-15T09:00:00.000Z" } } ``` ``` -------------------------------- ### Apify Client - Python Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Use the Apify Python client to integrate scraping APIs into Python applications. This example shows how to scrape LinkedIn job postings. ```APIDOC ## Apify Client - Python ### Description Integrates Apify scraping APIs into Python applications using the official Apify Python client library. ### Method N/A (Programmatic) ### Endpoint N/A (Uses Apify Client library) ### Parameters - **token** (string) - Required - Your Apify API token, used to initialize the `ApifyClient`. - **actorId** (string) - Required - The ID of the actor to run (e.g., `louisdeconinck/ai-job-search-agent`). - **runInput** (object) - Required - Input parameters for the actor run. - **searchUrl** (string) - The URL for the job search. - **maxItems** (integer) - Maximum number of job items to retrieve. - **proxy** (object) - Proxy configuration. - **useApifyProxy** (boolean) - Whether to use Apify Proxy. - **apifyProxyGroups** (array of strings) - List of proxy groups to use. ### Request Example ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") def scrape_linkedin_jobs(): # Configure the job scraper run_input = { "searchUrl": "https://www.linkedin.com/jobs/search/?keywords=software%20engineer&location=San%20Francisco", "maxItems": 100, "proxy": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"] } } # Run the LinkedIn Jobs Scraper run = client.actor("louisdeconinck/ai-job-search-agent").call(run_input=run_input) # Get results dataset = client.dataset(run["defaultDatasetId"]) items = list(dataset.iterate_items()) for job in items: print(f"Title: {job.get('title')}") print(f"Company: {job.get('company')}") print(f"Location: {job.get('location')}") print(f"Salary: {job.get('salary', 'Not specified')}") print("---") return items jobs = scrape_linkedin_jobs() ``` ### Response #### Success Response - **items** (list) - A list containing the scraped job data. - **title** (string) - The job title. - **company** (string) - The name of the hiring company. - **location** (string) - The job location. - **salary** (string) - The salary for the job (if available). #### Response Example ```python # Console output example: # Title: Software Engineer # Company: Tech Corp # Location: San Francisco, CA # Salary: $120,000 - $150,000 # --- # ... ``` ``` -------------------------------- ### Run Apify API with Python Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Employs the Apify Python client to integrate a LinkedIn Jobs Scraper into a Python application. This example shows how to configure the scraper with search criteria, proxy settings, execute the actor, and iterate through the returned job listings. ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") def scrape_linkedin_jobs(): # Configure the job scraper run_input = { "searchUrl": "https://www.linkedin.com/jobs/search/?keywords=software%20engineer&location=San%20Francisco", "maxItems": 100, "proxy": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"] } } # Run the LinkedIn Jobs Scraper run = client.actor("louisdeconinck/ai-job-search-agent").call(run_input=run_input) # Get results dataset = client.dataset(run["defaultDatasetId"]) items = list(dataset.iterate_items()) for job in items: print(f"Title: {job.get('title')}") print(f"Company: {job.get('company')}") print(f"Location: {job.get('location')}") print(f"Salary: {job.get('salary', 'Not specified')}") print("---") return items jobs = scrape_linkedin_jobs() ``` -------------------------------- ### GET /v2/actor-runs/{runId}/dataset/items Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Retrieve the output dataset from a completed API run using the dataset endpoint. This example shows how to get results from a completed run. ```APIDOC ## GET /v2/actor-runs/{runId}/dataset/items ### Description Retrieves the items from the dataset of a completed Apify Actor run. ### Method GET ### Endpoint `https://api.apify.com/v2/actor-runs/{runId}/dataset/items?token=YOUR_APIFY_TOKEN&format=json` ### Parameters #### Path Parameters - **runId** (string) - Required - The ID of the actor run from which to retrieve dataset items. #### Query Parameters - **token** (string) - Required - Your Apify API token. - **format** (string) - Optional - The desired format for the output (e.g., `json`, `csv`, `xml`). Defaults to `json`. ### Request Example ```bash curl "https://api.apify.com/v2/actor-runs/abc123xyz/dataset/items?token=YOUR_APIFY_TOKEN&format=json" ``` ### Response #### Success Response (200) - An array of objects, where each object represents a scraped item. - **title** (string) - Example field for a product title. - **price** (string) - Example field for a product price. - **rating** (number) - Example field for a product rating. - **reviews** (integer) - Example field for the number of reviews. - **asin** (string) - Example field for the Amazon Standard Identification Number. - **url** (string) - The URL from which the item was scraped. #### Response Example ```json [ { "title": "Product Name", "price": "$29.99", "rating": 4.5, "reviews": 1523, "asin": "B09V3KXJPB", "url": "https://amazon.com/dp/B09V3KXJPB" } ] ``` ``` -------------------------------- ### Apify Client - JavaScript/Node.js Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Use the Apify Client library to programmatically execute APIs and retrieve results in JavaScript/Node.js applications. This example shows how to scrape Google Maps data. ```APIDOC ## Apify Client - JavaScript/Node.js ### Description Integrates Apify scraping APIs into JavaScript/Node.js applications using the official Apify Client library. ### Method N/A (Programmatic) ### Endpoint N/A (Uses Apify Client library) ### Parameters - **token** (string) - Required - Your Apify API token, used to initialize the `ApifyClient`. - **actorId** (string) - Required - The ID of the actor to run (e.g., `raizen/ai-google-maps-extractor`). - **runInput** (object) - Optional - Input parameters for the actor run. - **searchQuery** (string) - Example input for Google Maps Extractor. - **maxResults** (integer) - Example input for Google Maps Extractor. - **language** (string) - Example input for Google Maps Extractor. - **includeReviews** (boolean) - Example input for Google Maps Extractor. ### Request Example ```javascript const { ApifyClient } = require('apify-client'); const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN', }); async function scrapeGoogleMaps() { // Run the AI Google Maps Extractor const run = await client.actor('raizen/ai-google-maps-extractor').call({ searchQuery: 'restaurants in New York', maxResults: 50, language: 'en', includeReviews: true }); // Fetch results from the run's dataset const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(`Found ${items.length} businesses:`); items.forEach(item => { console.log(`- ${item.name}: ${item.rating} stars, ${item.reviewCount} reviews`); console.log(` Address: ${item.address}`); console.log(` Phone: ${item.phone}`); }); return items; } scrapeGoogleMaps(); ``` ### Response #### Success Response - **items** (array) - An array containing the scraped data from the actor's default dataset. - **name** (string) - Name of the business. - **rating** (number) - Star rating of the business. - **reviewCount** (integer) - Number of reviews for the business. - **address** (string) - Address of the business. - **phone** (string) - Phone number of the business. #### Response Example ```javascript // Console output example: // Found 50 businesses: // - Restaurant Name: 4.5 stars, 1200 reviews // Address: 123 Main St, New York, NY // Phone: (212) 555-1234 // ... ``` ``` -------------------------------- ### Schedule Automated Apify API Runs Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Details how to create scheduled tasks for Apify APIs using the scheduling API. This example sets up a daily run of an Amazon search scraper at 9 AM UTC, specifying the actor to run and its input parameters. ```bash # Create a scheduled task to run daily at 9 AM UTC curl -X POST "https://api.apify.com/v2/schedules?token=YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "daily-price-monitor", "cronExpression": "0 9 * * *", "isEnabled": true, "isExclusive": true, "actions": [{ "type": "RUN_ACTOR", "actorId": "scraper-engine~amazon-search-scraper", "runInput": { "searchQuery": "laptop", "maxItems": 50, "sortBy": "price-asc" } }] }' # Response # { # "data": { # "id": "schedule123", # "name": "daily-price-monitor", # "cronExpression": "0 9 * * *", # "nextRunAt": "2025-01-15T09:00:00.000Z" # } # } ``` -------------------------------- ### Fetch Apify API Results via REST Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Shows how to retrieve the output data from a completed Apify API run using the dataset endpoint. It requires the run ID and API token, and allows specifying the desired output format (e.g., JSON). ```bash # Get results from a completed run curl "https://api.apify.com/v2/actor-runs/abc123xyz/dataset/items?token=YOUR_APIFY_TOKEN&format=json" # Response contains scraped data # [ # { # "title": "Product Name", # "price": "$29.99", # "rating": 4.5, # "reviews": 1523, # "asin": "B09V3KXJPB", # "url": "https://amazon.com/dp/B09V3KXJPB" # } # ] ``` -------------------------------- ### Monitor Amazon Product Prices with JavaScript Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt This JavaScript code monitors the prices of specific Amazon products using the Apify client. It takes a list of ASINs as input and returns detailed product information including price, discount, and rating. Ensure you have an Apify API token configured. ```javascript const { ApifyClient } = require('apify-client'); const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' }); async function monitorAmazonPrices(asins) { const results = []; for (const asin of asins) { const run = await client.actor('api-empire/amazon-asin-scraper').call({ asins: [asin], marketplace: 'US' }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); if (items.length > 0) { const product = items[0]; results.push({ asin: asin, title: product.title, currentPrice: product.price, originalPrice: product.originalPrice, discount: product.discount, rating: product.rating, availability: product.availability, timestamp: new Date().toISOString() }); console.log(`${product.title}: ${product.price}`); if (product.discount) { console.log(` Discount: ${product.discount}% off!`); } } } return results; } // Monitor specific products const productsToTrack = ['B09V3KXJPB', 'B08N5WRWNW', 'B09B8YWWSX']; monitorAmazonPrices(productsToTrack).then(prices => { // Store prices in database or send alerts console.log('Price monitoring complete:', prices.length, 'products tracked'); }); ``` -------------------------------- ### Fetching All Available APIs from Apify Store Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Use the Apify Store API to fetch metadata about all available scraping APIs. This endpoint allows you to discover and list actors available in the Apify Store. ```APIDOC ## GET /v2/store ### Description Fetches metadata for all available actors (scraping APIs) from the Apify Store. You can paginate the results using `limit` and `offset` parameters. ### Method GET ### Endpoint /v2/store ### Parameters #### Query Parameters - **limit** (integer) - Optional - The maximum number of actors to return per page. Defaults to 100. - **offset** (integer) - Optional - The number of actors to skip before starting to collect the result set. Defaults to 0. ### Response #### Success Response (200) - **data** (object) - Contains the list of actors and total count. - **total** (integer) - The total number of actors available in the store. - **items** (array) - An array of actor objects. - **name** (string) - The unique name of the actor. - **title** (string) - The display title of the actor. - **description** (string) - A brief description of the actor's functionality. - **username** (string) - The username of the actor's owner. - **categories** (array) - An array of categories the actor belongs to. - **stats** (object) - Statistics about the actor's usage. #### Response Example ```json { "data": { "total": 1500, "items": [ { "name": "twitter-scraper", "title": "Twitter Scraper", "description": "Scrapes tweets and user data from Twitter.", "username": "apify", "categories": ["Social Media", "Web Scraping"], "stats": {"runs": 10000, "failedRuns": 50} } // ... more actors ] } } ``` ``` -------------------------------- ### Generate Business Leads with Email Extraction using Python Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt This Python script generates business leads by extracting contact information, including emails, from Google Maps. It takes an industry and location as input and returns a list of businesses with their details. Requires an Apify API token. ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") def find_business_leads(industry, location, max_leads=100): """ Find business leads with contact information """ # First, search Google Maps for businesses maps_input = { "searchQuery": f"{industry} in {location}", "maxResults": max_leads, "includeWebsite": True, "includeEmail": True } run = client.actor("raizen/ai-google-maps-extractor").call(run_input=maps_input) dataset = client.dataset(run["defaultDatasetId"]) businesses = list(dataset.iterate_items()) leads = [] for biz in businesses: lead = { "name": biz.get("name"), "address": biz.get("address"), "phone": biz.get("phone"), "website": biz.get("website"), "email": biz.get("email"), "rating": biz.get("rating"), "reviews": biz.get("reviewCount") } leads.append(lead) print(f"Lead: {lead['name']}") print(f" Contact: {lead['phone']} | {lead['email']}") print(f" Website: {lead['website']}") return leads # Find restaurant leads in Los Angeles leads = find_business_leads("restaurants", "Los Angeles", max_leads=50) print(f"\nFound {len(leads)} business leads") ``` -------------------------------- ### Configure Webhook for API Run Completion Notifications (JavaScript) Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Sets up an Express.js server to listen for webhook events from Apify. It specifically handles the 'ACTOR.RUN.SUCCEEDED' event to process results from completed API runs. Requires the 'express' and 'apify-client' packages. ```javascript const express = require('express'); const app = express(); app.use(express.json()); // Webhook endpoint to receive Apify notifications app.post('/webhook/apify', (req, res) => { const { eventType, eventData } = req.body; if (eventType === 'ACTOR.RUN.SUCCEEDED') { const { actorRunId, defaultDatasetId } = eventData; console.log(`Run ${actorRunId} completed successfully`); console.log(`Dataset ID: ${defaultDatasetId}`); // Process the results processScrapedData(defaultDatasetId); } res.status(200).send('OK'); }); async function processScrapedData(datasetId) { const { ApifyClient } = require('apify-client'); const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' }); const { items } = await client.dataset(datasetId).listItems(); console.log(`Processing ${items.length} items...`); // Store in database, send notifications, etc. } app.listen(3000); ``` -------------------------------- ### Fetch All Available APIs from Apify Store (JavaScript) Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Fetches metadata for all available scraping APIs from the Apify Store using the official API. It makes an HTTPS request and parses the JSON response to extract actor details. This function can be configured with limit and offset parameters. ```javascript const https = require('https'); async function fetchApifyActors(limit = 100, offset = 0) { return new Promise((resolve, reject) => { const options = { hostname: 'api.apify.com', path: `/v2/store?limit=${limit}&offset=${offset}`, method: 'GET', headers: { 'User-Agent': 'Mozilla/5.0' } }; const req = https.request(options, (res) => { let data = ''; res.on('data', chunk => data += chunk); res.on('end', () => { const json = JSON.parse(data); const actors = json.data.items.map(actor => ({ name: actor.name, title: actor.title, description: actor.description, url: `https://apify.com/${actor.username}/${actor.name}`, categories: actor.categories || [], stats: actor.stats })); resolve({ actors, total: json.data.total }); }); }); req.on('error', reject); req.end(); }); } // Fetch first 100 APIs fetchApifyActors(100, 0).then(({ actors, total }) => { console.log(`Total APIs available: ${total}`); actors.forEach(a => console.log(`- ${a.title}: ${a.url}`)); }); ``` -------------------------------- ### Run Apify API with JavaScript/Node.js Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Utilizes the Apify Client library for Node.js to programmatically execute a Google Maps Extractor API and process its results. It demonstrates authenticating, calling the actor with specific input parameters, and fetching the scraped data from the dataset. ```javascript const { ApifyClient } = require('apify-client'); const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN', }); async function scrapeGoogleMaps() { // Run the AI Google Maps Extractor const run = await client.actor('raizen/ai-google-maps-extractor').call({ searchQuery: 'restaurants in New York', maxResults: 50, language: 'en', includeReviews: true }); // Fetch results from the run's dataset const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(`Found ${items.length} businesses:`); items.forEach(item => { console.log(`- ${item.name}: ${item.rating} stars, ${item.reviewCount} reviews`); console.log(` Address: ${item.address}`); console.log(` Phone: ${item.phone}`); }); return items; } scrapeGoogleMaps(); ``` -------------------------------- ### Scrape Twitter Posts using Apify Client (Python) Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Demonstrates how to scrape tweets from X (formerly Twitter) using the Apify client library and a specialized Twitter scraper actor. It allows specifying search terms and the maximum number of tweets to retrieve. Results are then iterated and printed. ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") def scrape_twitter_posts(search_query, max_tweets=100): """ Scrape tweets using the X/Twitter scraper API """ run_input = { "searchTerms": [search_query], "maxTweets": max_tweets, "includeReplies": False, "onlyVerified": False, "proxy": { "useApifyProxy": True } } # Run the Twitter scraper run = client.actor("mikolabs/x-scraper").call(run_input=run_input) # Process results dataset = client.dataset(run["defaultDatasetId"]) tweets = list(dataset.iterate_items()) for tweet in tweets: print(f"@{tweet.get('username')}: {tweet.get('text')[:100]}...") print(f" Likes: {tweet.get('likes')} | Retweets: {tweet.get('retweets')}") print(f" Date: {tweet.get('createdAt')}") return tweets # Example: Scrape tweets about AI tweets = scrape_twitter_posts("artificial intelligence", max_tweets=50) ``` -------------------------------- ### Scrape Real Estate Listings for Market Analysis with JavaScript Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt This JavaScript code scrapes real estate listings for a given location and property type (e.g., 'sale'). It performs basic market analysis, calculating average price, price range, and listing counts by bedrooms. Requires an Apify API token. ```javascript const { ApifyClient } = require('apify-client'); const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' }); async function scrapeRealEstateListings(location, propertyType = 'sale') { const run = await client.actor('harvestlabs/ai-real-estate-agent').call({ query: `${propertyType} properties in ${location}`, maxListings: 100, includeImages: true, includeDetails: true }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const analysis = { totalListings: items.length, averagePrice: 0, priceRange: { min: Infinity, max: 0 }, byBedrooms: {} }; let totalPrice = 0; items.forEach(listing => { const price = parseFloat(listing.price?.replace(/[^0-9.]/g, '') || 0); if (price > 0) { totalPrice += price; analysis.priceRange.min = Math.min(analysis.priceRange.min, price); analysis.priceRange.max = Math.max(analysis.priceRange.max, price); } const beds = listing.bedrooms || 'unknown'; analysis.byBedrooms[beds] = (analysis.byBedrooms[beds] || 0) + 1; console.log(`${listing.address}: ${listing.price}`); console.log(` ${listing.bedrooms} bed, ${listing.bathrooms} bath`); }); analysis.averagePrice = totalPrice / items.filter(i => i.price).length; console.log('\n--- Market Analysis ---'); console.log(`Average Price: $${analysis.averagePrice.toLocaleString()}`); console.log(`Price Range: $${analysis.priceRange.min.toLocaleString()} - $${analysis.priceRange.max.toLocaleString()}`); return { listings: items, analysis }; } scrapeRealEstateListings('Miami, FL', 'sale'); ``` -------------------------------- ### Webhook Integration Source: https://context7.com/cporter202/scraping-apis-for-devs/llms.txt Configure webhooks to receive notifications when API runs complete. This endpoint listens for 'ACTOR.RUN.SUCCEEDED' events and processes the scraped data. ```APIDOC ## POST /webhook/apify ### Description This endpoint receives notifications from Apify when an actor run completes successfully. It logs the run details and triggers a function to process the scraped data from the default dataset. ### Method POST ### Endpoint /webhook/apify ### Parameters #### Request Body - **eventType** (string) - Required - The type of event, e.g., 'ACTOR.RUN.SUCCEEDED'. - **eventData** (object) - Required - Data related to the event. - **actorRunId** (string) - Required - The ID of the actor run. - **defaultDatasetId** (string) - Required - The ID of the dataset associated with the run. ### Request Example ```json { "eventType": "ACTOR.RUN.SUCCEEDED", "eventData": { "actorRunId": "some_run_id", "defaultDatasetId": "some_dataset_id" } } ``` ### Response #### Success Response (200) - **message** (string) - Confirmation message, e.g., 'OK'. #### Response Example ```json "OK" ``` ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.