### Setup Request Queue and Dataset Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Initializes the request queue, adds the starting URL, and opens the dataset for result storage. ```javascript const requestQueue = await Apify.openRequestQueue(); await requestQueue.addRequest({ url: 'https://trends.google.com/trends', userData: { label: 'START' }, }); const dataset = await Apify.openDataset(); let { itemCount } = await dataset.getInfo(); ``` -------------------------------- ### Initialize proxy configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Example usage of the proxyConfiguration function to set up proxy parameters. ```javascript const config = await proxyConfiguration({ proxyConfig: input.proxyConfiguration, required: true, blacklist: ['GOOGLESERP'], hint: ['RESIDENTIAL'] }); ``` -------------------------------- ### Handle START Request Label Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Enqueues search request sources created from input when the label is 'START'. ```javascript if (label === 'START') { for (const source of sources) { await requestQueue.addRequest(source); } } ``` -------------------------------- ### Example RequestSource Object Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/types.md A sample object structure for a search request. ```javascript { url: "https://trends.google.com/trends/explore?geo=US&date=today%201-m&cat=5&q=web%20scraping", userData: { label: 'SEARCH' } } ``` -------------------------------- ### Example ProxyConfig Objects Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/types.md Sample configurations for using Apify Proxy or custom proxy lists. ```javascript { useApifyProxy: true } ``` ```javascript { proxyUrls: [ "http://proxy1.example.com:8080", "http://proxy2.example.com:8080" ] } ``` -------------------------------- ### Complete Input Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Example of a fully populated JSON input object for the scraper. ```json { "searchTerms": [ "web scraping", "python", "javascript" ], "timeRange": "today 1-m", "category": "5", "geo": "US", "maxItems": 1000, "outputAsISODate": true, "maxConcurrency": 10, "pageLoadTimeoutSecs": 180, "extendOutputFunction": "($) => { return { customField: 'value' }; }", "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Configure Input JSON Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/README.md Example configuration object for the scraper, including search terms, time ranges, and proxy settings. ```jsonc { "searchTerms": [ "test term", "another test term" ], "spreadsheetId": //spreadsheetId, "timeRange": "now 4-H", "isPublic": true, "maxItems": 100, "customTimeRange": "2020-03-24 2020-03-29", "extendOutputFunction": "($) => { return { test: 1234, test2: 5678 } }", "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Define and Implement ExtendOutputFunction Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/types.md Defines the structure for custom output extension functions and provides an example implementation using jQuery. ```javascript type ExtendOutputFunction = ($: JQuery) => object ``` ```javascript ($) => { return { comment: 'Custom field', trends: $('a[href^="/trends/explore"]').map((_, el) => ({ text: el.textContent, url: el.href })).toArray() } } ``` -------------------------------- ### Scraping Result Output Examples Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/types.md Various formats of the scraping result object, including standard, ISO date, extended, and no-data scenarios. ```javascript { "Term / Date": "web scraping", "Jan 13, 2019": "92", "Jan 20, 2019": "100", "Jan 27, 2019": "86" } ``` ```javascript { "Term / Date": "web scraping", "2019-01-13T00:00:00.000Z": "92", "2019-01-20T00:00:00.000Z": "100", "2019-01-27T00:00:00.000Z": "86" } ``` ```javascript { "Term / Date": "web scraping", "2019-01-13T00:00:00.000Z": "92", "2019-01-20T00:00:00.000Z": "100", "customField": "custom value", "trends": [ { text: "python web scraping", link: "https://..." }, { text: "web scraping tools", link: "https://..." } ] } ``` ```javascript { "Term / Date": "obscure_search_term_12345", "message": "The search term displays no data." } ``` -------------------------------- ### Define and Example SheetImportResult Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/types.md Defines the structure for Google Sheets import results and provides a sample data object. ```javascript type SheetImportResult = { output: { body: Array<{[columnName: string]: string}> } } ``` ```javascript { output: { body: [ { "Search Terms": "web scraping" }, { "Search Terms": "python" }, { "Search Terms": "javascript" } ] } } ``` -------------------------------- ### Exported CSV Data Format Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Example of the CSV structure generated when exporting results from the Apify dataset. ```csv Term / Date,Jan 13 2019,Jan 20 2019,Jan 27 2019 python,92,100,86 javascript,88,95,82 typescript,45,52,49 ``` -------------------------------- ### Exported JSON Data Format Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Example of the JSON structure generated when exporting results from the Apify dataset. ```json [ { "Term / Date": "python", "Jan 13, 2019": "92", "Jan 20, 2019": "100" }, { "Term / Date": "javascript", "Jan 13, 2019": "88", "Jan 20, 2019": "95" } ] ``` -------------------------------- ### Validate Extend Output Function Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Ensures the provided extend output function is valid before starting the crawler. ```javascript checkAndEval(extendOutputFunction); ``` -------------------------------- ### ISO Date Output Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Configuration to toggle between standard and ISO date formats in the output. ```json { "searchTerms": ["nft"], "outputAsISODate": true, "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "Term / Date": "nft", "Jan 13, 2021": "92", "Jan 20, 2021": "100" } ``` ```json { "Term / Date": "nft", "2021-01-13T00:00:00.000Z": "92", "2021-01-20T00:00:00.000Z": "100" } ``` -------------------------------- ### Apify Platform Deployment Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Command to push the actor to the Apify platform. ```bash apify push ``` -------------------------------- ### Minimum Viable Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md The simplest valid configuration required to run the actor. ```json { "searchTerms": ["example"], "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Usage of BASE_URL in URL construction Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/constants.md Demonstrates importing the constant and using it to build a search URL with query parameters. ```javascript const { BASE_URL } = require('./constants'); // BASE_URL is used internally by newUrl(): const url = new URL(BASE_URL); url.searchParams.set('q', 'web scraping'); url.searchParams.set('geo', 'US'); // Result: https://trends.google.com/trends/explore?q=web+scraping&geo=US ``` -------------------------------- ### Log Spreadsheet Instructions Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Logs instructions for accessing results as a spreadsheet if a spreadsheet ID is provided in the input. ```javascript if (spreadsheetId) { log.info(` You can get the results as usual, or download them in spreadsheet format under the 'dataset' tab of your actor run. More information in the actor documentation. `); } ``` -------------------------------- ### Configure Proxy Settings Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/errors.md Provide valid proxy configuration when running on the Apify platform to avoid missing proxy errors. ```javascript // On Apify platform, provide proxy config: { "proxyConfiguration": { "useApifyProxy": true } } // Or custom proxies: { "proxyConfiguration": { "proxyUrls": ["http://proxy:port"] } } ``` -------------------------------- ### Configure Proxy Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Initializes and validates the proxy settings for the crawler. ```javascript const proxyConfig = await proxyConfiguration({ proxyConfig: input.proxyConfiguration, }); ``` -------------------------------- ### Project Directory Structure Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/README.md Visual representation of the project file organization and core source files. ```text actor-google-trends-scraper/ ├── src/ │ ├── main.js # Crawler orchestration │ ├── utils.js # Utility functions │ └── constants.js # Constants ├── INPUT_SCHEMA.json # Input schema definition ├── apify.json # Actor configuration ├── package.json # Dependencies └── README.md # Project overview ``` -------------------------------- ### Define Actor Entry Point Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Uses Apify.main to encapsulate the asynchronous execution logic of the actor. ```javascript Apify.main(async () => { // Initialization and crawler setup // Request handling // Dataset management }); ``` -------------------------------- ### proxyConfiguration(params) Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Creates and validates proxy configuration for the Apify crawler, ensuring compliance with platform requirements and blacklist constraints. ```APIDOC ## proxyConfiguration(params) ### Description Creates and validates proxy configuration for the Apify crawler. It handles proxy object creation, platform-specific validation, and blacklist enforcement. ### Parameters - **proxyConfig** (any) - Optional - Proxy configuration object - **required** (boolean) - Optional - Whether proxy is required when running on Apify platform (Default: true) - **force** (boolean) - Optional - Force proxy validation regardless of environment (Default: Apify.isAtHome()) - **blacklist** (string[]) - Optional - Proxy groups to reject (Default: ['GOOGLESERP']) - **hint** (string[]) - Optional - Suggested proxy groups to use (Default: []) ### Return Type Promise - Configured proxy object or undefined ### Throws - **Error**: Proxy is required but not provided when on Apify platform - **Error**: Blacklisted proxy group is configured ### Example ```javascript const config = await proxyConfiguration({ proxyConfig: input.proxyConfiguration, required: true, blacklist: ['GOOGLESERP'], hint: ['RESIDENTIAL'] }); ``` ``` -------------------------------- ### Configure Search Terms Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Provide a list of search terms to be scraped. ```json { "searchTerms": ["web scraping", "python", "javascript"] } ``` -------------------------------- ### Configure proxyConfiguration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Sets proxy settings for the crawler. Apify Proxy is used by default. ```json { "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "proxyConfiguration": { "useApifyProxy": false } ``` ```json { "proxyConfiguration": { "proxyUrls": [ "http://proxy1.example.com:8080", "http://proxy2.example.com:8080" ] } } ``` -------------------------------- ### Configure outputAsISODate Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Controls the date format in the output. Enabling this converts dates to ISO 8601 format. ```json { "searchTerm": "CNN", "Jan 13, 2019": "92", "Jan 20, 2019": "100" } ``` ```json { "searchTerm": "CNN", "2019-01-13T00:00:00.000Z": "92", "2019-01-20T00:00:00.000Z": "100" } ``` -------------------------------- ### Initialize Apify Actor Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/overview.md The main entry point for the actor, which retrieves input and orchestrates the crawler lifecycle. ```javascript Apify.main(async () => { const input = await Apify.getInput(); // ... crawler initialization and execution }); ``` -------------------------------- ### Implement handlePageFunction Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md The primary entry point for processing pages, including rate limit checks and request routing based on labels. ```javascript handlePageFunction: async ({ page, request }) => { // Check max items limit if (maxItems && maxItemsCheck(maxItems, itemCount)) { return; } // Check for 429 error const is429 = await page.evaluate(() => !!document.querySelector('div#af-error-container')); if (is429) { await Apify.utils.sleep(10000); throw 'Page got a 429 Error...'; } const { label } = request.userData; if (label === 'START') { // Hot start - enqueue actual search requests for (const source of sources) { await requestQueue.addRequest(source); } } else if (label === 'SEARCH') { // Process search results } } ``` -------------------------------- ### Retrieve and Validate Input Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Fetches the actor configuration and ensures it meets schema requirements. ```javascript const input = await Apify.getInput(); validateInput(input); ``` -------------------------------- ### Define proxyConfiguration function signature Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Defines the input parameters and return type for the proxy configuration utility. ```javascript proxyConfiguration(params: { proxyConfig?: any, required?: boolean, force?: boolean, blacklist?: string[], hint?: string[] }): Promise ``` -------------------------------- ### Configure Time Range Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Set a predefined time range for the search results. ```json { "timeRange": "today 1-m" } ``` -------------------------------- ### Configure maxConcurrency Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Sets the maximum number of pages to open in parallel. Higher values increase resource usage. ```json { "maxConcurrency": 5 } ``` -------------------------------- ### Handle No Data Case Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Creates an output object with a no-data message and pushes it to the dataset if no trend data is found. ```javascript if (!hasData) { const resObject = Object.create(null); resObject[sheetTitle] = searchTerm; resObject.message = 'The search term displays no data.'; const result = await applyFunction(page, extendOutputFunction); await Apify.pushData({ ...resObject, ...result }); // Save snapshot and log } ``` -------------------------------- ### Execute Crawler Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Initiates the crawler process and waits for the operation to complete. ```javascript log.info('Starting crawler.'); await crawler.run(); log.info('Crawler Finished.'); ``` -------------------------------- ### Initialize PuppeteerCrawler Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Configures the crawler with session pooling, proxy settings, and concurrency limits. ```javascript const crawler = new Apify.PuppeteerCrawler({ requestQueue, maxRequestRetries: 3, handlePageTimeoutSecs: 300, maxConcurrency, useSessionPool: true, sessionPoolOptions: { maxPoolSize: maxConcurrency, sessionOptions: { maxErrorScore: 5, }, }, proxyConfiguration: proxyConfig, browserPoolOptions: { maxOpenPagesPerBrowser: 1, }, persistCookiesPerSession: true, preNavigationHooks: [...], postNavigationHooks: [...], handlePageFunction: async ({ page, request }) => { ... }, handleFailedRequestFunction: async ({ request }) => { ... } }); ``` -------------------------------- ### Bulk Search Input Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Recommended input structure for processing large batches of search terms to optimize costs. ```json { "searchTerms": [ "term1", "term2", ... "term1000" ], "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Manage Session Reuse Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Demonstrates how session pools maintain cookies across multiple requests to improve efficiency. ```javascript Session Pool: Session 1 (with cookies) → Request 1, 2, 3 Session 2 (with cookies) → Request 4, 5, 6 Session 3 (with cookies) → Request 7, 8, 9 (Each session maintains cookies across requests) ``` -------------------------------- ### Configure maxItems Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Sets the maximum number of search results to scrape. A value of 0 disables the limit. ```json { "maxItems": 100 } ``` -------------------------------- ### Custom Output Function - Add Field Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Configuration to inject custom fields into the output using a JavaScript function. ```json { "searchTerms": ["web scraping"], "extendOutputFunction": "($) => { return { source: 'google_trends', scraped_at: new Date().toISOString() }; }", "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "Term / Date": "web scraping", "Jan 13, 2019": "92", "source": "google_trends", "scraped_at": "2024-11-13T10:30:00.000Z" } ``` -------------------------------- ### Custom Function Application Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Logic for applying custom output functions in src/utils.js. ```javascript extendOutputFunction provided? → { Yes ├─ Inject jQuery into page ├─ Evaluate function with $ parameter ├─ Validate returns object ├─ Catch errors, log warning └─ Return customOutput object No └─ Return empty object {} } mergedOutput = {...defaultOutput, ...customOutput} ``` -------------------------------- ### Configure pageLoadTimeoutSecs Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Sets the timeout in seconds for page loading. Minimum recommended value is 30 seconds. ```json { "pageLoadTimeoutSecs": 240 } ``` -------------------------------- ### Complete Actor Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/README.md A comprehensive JSON object defining search parameters, performance limits, and custom output functions for the scraper. ```json { "searchTerms": ["web scraping", "python"], "timeRange": "today 1-m", "geo": "US", "category": "5", "maxItems": 1000, "outputAsISODate": true, "maxConcurrency": 10, "pageLoadTimeoutSecs": 180, "extendOutputFunction": "($) => { return { custom: 'field' }; }", "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Integrate Google Sheets Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Imports search terms from Google Sheets. Requires specific authorization steps for private sheets. ```json { "spreadsheetId": "1mZq8Z_q7K_r8Y_t7K_q1Z_w2X_y3Z_a4B_c5D_e6F_g7H", "isPublic": true, "timeRange": "today 1-m", "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "spreadsheetId": "1mZq8Z_q7K_r8Y_t7K_q1Z_w2X_y3Z_a4B_c5D_e6F_g7H", "isPublic": false, "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Handle Search Terms with No Data Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Demonstrates how the actor handles terms that return no results without failing the entire run. ```json { "searchTerms": ["obscure_term_xyz_123"] } ``` ```json { "Term / Date": "obscure_term_xyz_123", "message": "The search term displays no data." } ``` -------------------------------- ### Format Date Strings to ISO Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Parses various date string formats into ISO 8601 strings when outputAsISODate is enabled. ```javascript Input: "Jan 13, 2019" Processing: 1. Try parsing as Date 2. If year < (currentYear - 5), add current year 3. Handle "Month Day at HH:MM AM/PM" format 4. Handle "HH:MM AM/PM" format 5. Return toISOString() Output: "2019-01-13T00:00:00.000Z" ``` -------------------------------- ### Optimize Memory Management Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Configures browser pool settings to balance memory usage against process isolation. ```javascript browserPoolOptions: maxOpenPagesPerBrowser: 1 Result: 10 concurrent pages = 10 browser instances (Higher memory usage but better isolation) ``` -------------------------------- ### Apply Custom Function and Push Data Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Executes the custom output function and pushes the final result object to the dataset. ```javascript const result = await applyFunction(page, extendOutputFunction); await Apify.pushData({ ...resObject, ...result }); itemCount++; ``` -------------------------------- ### ISO Date Output Format Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/README.md Output structure when the outputAsISODate option is enabled. ```jsonc { "Term / Date": "CNN", "2019-08-11T03:00:00.000Z": "43", "2019-08-18T03:00:00.000Z": "34", "2019-08-25T03:00:00.000Z": "34", // ... } ``` -------------------------------- ### Define Post-Navigation Hooks Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Ensures the page is brought to the front after navigation completes. ```javascript postNavigationHooks: [async ({ page }) => { await page.bringToFront(); }] ``` -------------------------------- ### Multiple Search Terms Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Configuration for scraping multiple terms over a 30-day period. ```json { "searchTerms": [ "python", "javascript", "typescript", "rust", "golang" ], "timeRange": "today 1-m", "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "Term / Date": "python", "2024-10-13": "95", "2024-10-20": "98", "2024-10-27": "92" } ``` -------------------------------- ### Handle SEARCH Request Label Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Initializes the search request handler by injecting jQuery if needed and parsing the search term from the URL. ```javascript } else if (label === 'SEARCH') { if (extendOutputFunction) { await Apify.utils.puppeteer.injectJQuery(page); } const queryStringObj = new URL(request.url); const searchTerm = queryStringObj.searchParams.get('q'); const terms = [...(searchTerm?.split(','))]; ``` -------------------------------- ### Basic Single Search Term Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Configuration for scraping a single term with default settings. ```json { "searchTerms": ["web scraping"], "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "Term / Date": "web scraping", "Jan 13, 2019": "92", "Jan 20, 2019": "100", "Jan 27, 2019": "86", ... } ``` -------------------------------- ### Local Execution Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Commands to run the actor locally. Requires an INPUT.json file in the working directory. ```bash cd /path/to/actor-google-trends-scraper npm install npm start ``` -------------------------------- ### Define Pre-Navigation Hooks Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Sets the navigation timeout and wait strategy before each page load. ```javascript preNavigationHooks: [async (context, gotoOptions) => { gotoOptions.timeout = pageLoadTimeoutSecs * 1000; gotoOptions.waitUntil = 'domcontentloaded'; }] ``` -------------------------------- ### Custom Output Function Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/README.md Input configuration to inject a custom JavaScript function for transforming the output data. ```json { "searchTerms": ["ai"], "extendOutputFunction": "($) => { return { source: 'trends' }; }", "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### validateInput(input: any): void Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Validates the actor's input configuration against required and type constraints. Throws an error if validation fails. ```APIDOC ## validateInput ### Description Validates the actor's input configuration against required and type constraints. If the input is invalid, it throws an Error. ### Signature `validateInput(input: any): void` ### Parameters - **input** (any) - Required - The input object to validate. ### Throws - **Error** - Thrown if INPUT is missing, required fields are missing, or if provided fields do not match expected types or formats (e.g., spreadsheetId length). ### Example ```javascript const input = { searchTerms: ['test', 'example'], timeRange: 'now 1-d', geo: 'US', category: '5', proxyConfiguration: { useApifyProxy: true } }; validateInput(input); // Throws if invalid, otherwise returns ``` ``` -------------------------------- ### Destructure Input Parameters Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Extracts configuration fields from the input object with default values applied. ```javascript const { searchTerms, spreadsheetId, isPublic = false, timeRange, category, maxItems = null, customTimeRange = null, geo = null, extendOutputFunction = null, pageLoadTimeoutSecs = 180, maxConcurrency = 10, outputAsISODate = false, } = input; ``` -------------------------------- ### Configure Maximum Item Limit Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Limits the number of search terms processed to manage costs. Set to 0 or omit for no limit. ```json { "searchTerms": [ "term1", "term2", "term3", "... (1000 terms) ..." ], "maxItems": 100, "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Create URL Sources Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Generates request URLs based on provided search terms or Google Sheet data. ```javascript const { sources, sheetTitle } = await checkAndCreateUrlSource( searchTerms, spreadsheetId, isPublic, timeRange, category, customTimeRange, geo, ); ``` -------------------------------- ### Standard Output Format Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/README.md The default structure of an output item containing search term data keyed by date. ```jsonc { "searchTerm": "CNN", "‪Jan 13, 2019‬": "92", "‪Jan 20, 2019‬": "100", "‪Jan 27, 2019‬": "86", "‪Feb 3, 2019‬": "82", //... } ``` -------------------------------- ### Page Processing Logic Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Logic for handling page navigation and task routing in src/main.js. ```javascript Page Loaded → { Check 429 error ↓ Extract label from userData ↓ label === 'START'? → Enqueue all SEARCH requests → Return ↓ label === 'SEARCH' → Continue to data extraction } ``` -------------------------------- ### Create Request Sources Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Generates an array of Apify request objects from search terms or a Google Sheet. Throws an error if the imported spreadsheet contains more than one column. ```javascript const { sources, sheetTitle } = await checkAndCreateUrlSource( ['python', 'javascript'], undefined, false, 'today 1-m', '5', null, 'US' ); console.log(sheetTitle); // 'Term / Date' console.log(sources.length); // 2 console.log(sources[0].userData.label); // 'SEARCH' ``` -------------------------------- ### Configure Custom Time Range Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Define a specific date or date-time range for the search. ```json { "customTimeRange": "2020-03-24 2020-03-29" } ``` ```json { "customTimeRange": "2020-03-24T08 2020-03-29T15" } ``` -------------------------------- ### Default Input Values Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md The default configuration applied when no input is provided on the Apify platform. ```json { "searchTerms": [], "isPublic": false, "timeRange": "", "maxItems": 0, "customTimeRange": null, "extendOutputFunction": null, "geo": null, "category": null, "outputAsISODate": false, "maxConcurrency": 10, "pageLoadTimeoutSecs": 180, "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Configure Public Sheet Access Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Set to true to allow importing public Google Sheets without authorization. ```json { "isPublic": true } ``` -------------------------------- ### Implement handleFailedRequestFunction Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/main.md Logs a warning and pushes debug information to the dataset when a request fails repeatedly. ```javascript handleFailedRequestFunction: async ({ request }) => { log.warning(`Request ${request.url} failed too many times`); await dataset.pushData({ '#debug': Apify.utils.createRequestDebugInfo(request), }); } ``` -------------------------------- ### Exported Utilities from utils.js Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/README.md Lists the utility functions exported from the utils module for input validation, URL generation, and proxy configuration. ```javascript module.exports = { validateInput, // Validate input object parseKeyAsIsoDate, // Convert dates to ISO format checkAndCreateUrlSource, // Generate URLs from terms or sheet proxyConfiguration, // Setup proxy configuration maxItemsCheck, // Check item limit checkAndEval, // Validate function string applyFunction, // Execute custom output function }; ``` -------------------------------- ### checkAndCreateUrlSource(searchTerms, spreadsheetId, isPublic, timeRange, category, customTimeRange, geo) Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Creates an array of request sources (URLs) from either direct search terms or a Google Sheet. ```APIDOC ## checkAndCreateUrlSource(...) ### Description Creates an array of request sources (URLs) from either direct search terms or a Google Sheet. ### Parameters - **searchTerms** (string[]) - Optional - Array of search terms to create URLs for - **spreadsheetId** (string) - Optional - Google Sheet ID to import search terms from - **isPublic** (boolean) - Optional - Whether the spreadsheet is public (default: false) - **timeRange** (string) - Optional - Default time range code - **category** (string) - Optional - Category code to filter by - **customTimeRange** (string) - Optional - Custom time range (takes precedence over timeRange) (default: null) - **geo** (string) - Optional - Geographic area code ### Return Type Promise<{ sources: Apify.RequestOptions[], sheetTitle: string }> ### Throws - **Error** - Thrown if the spreadsheet has more than one column. ### Example ```javascript const { sources, sheetTitle } = await checkAndCreateUrlSource( ['python', 'javascript'], undefined, false, 'today 1-m', '5', null, 'US' ); ``` ``` -------------------------------- ### Custom Time Range Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Configuration for specific date ranges or high-precision time intervals. ```json { "searchTerms": ["covid-19"], "customTimeRange": "2020-01-01 2020-12-31", "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "customTimeRange": "2024-11-01T08 2024-11-07T18" } ``` -------------------------------- ### Output Object Construction Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Mapping of extracted data into the final result object. ```javascript resObject = { [sheetTitle]: searchTerm, // e.g., "Term / Date": "python" [date1]: value1, // e.g., "Jan 13, 2019": "92" [date2]: value2, // e.g., "Jan 20, 2019": "100" ... } For multiple search terms in one query: values = term1_value, term2_value, term3_value ``` -------------------------------- ### Configure Parallel Processing Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Defines the concurrency limit for requests and illustrates the sequential execution timeline. ```javascript maxConcurrency: 10 (default) Timeline: 0ms: Request 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 start 5000ms: Request 1 completes → Request 11 starts 10000ms: Request 2 completes → Request 12 starts ... ``` -------------------------------- ### Resolve Blacklisted Proxy Group Errors Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/errors.md Avoid using the blacklisted GOOGLESERP proxy group by using default settings or alternative groups. ```javascript // Don't use GOOGLESERP group // Instead, in Apify proxy settings: // 1. Use default Apify Proxy (no specific group) // 2. Or select RESIDENTIAL or other non-blacklisted group // 3. Contact Apify support if you need GOOGLESERP access { "proxyConfiguration": { "useApifyProxy": true // Don't specify groups, use defaults } } ``` -------------------------------- ### Execute custom output function with applyFunction Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Injects jQuery into the page context to evaluate a custom function string. Returns an empty object if the input is null or if the function execution fails. ```javascript const result = await applyFunction(page, '($) => { return { custom: "field" }; }'); // result = { custom: 'field' } const result2 = await applyFunction(page, null); // result2 = {} ``` -------------------------------- ### Fix Invalid proxyConfiguration Type Error Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/errors.md Ensure proxyConfiguration is provided as an object. ```json // Incorrect: { "proxyConfiguration": "useApifyProxy=true" } // Correct: { "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Extend output with custom fields Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/README.md Add or modify output data by returning an object from a function that receives a jQuery handle. ```js ($) => { return { comment: 'This is a comment', } } ``` ```js ($) => { return { trends: $('a[href^="/trends/explore"] .label-text') .map((_, s) => ({ text: s.innerText, link: s.closest('a').href })) .toArray() } } ``` -------------------------------- ### Exported Utility Functions Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md List of functions exported from the utility module for use in other parts of the application. ```javascript module.exports = { validateInput, parseKeyAsIsoDate, checkAndCreateUrlSource, proxyConfiguration, maxItemsCheck, checkAndEval, applyFunction, }; ``` -------------------------------- ### parseKeyAsIsoDate(key) Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Parses various date string formats from Google Trends output into a standard ISO 8601 format. ```APIDOC ## parseKeyAsIsoDate(key) ### Description Parses date strings from Google Trends output into ISO 8601 format. Handles multiple formats including month-day, time-only, and specific date-time strings. ### Parameters - **key** (string) - Required - Date string from Google Trends ### Return Type string - ISO 8601 formatted date string ### Example ```javascript parseKeyAsIsoDate('Jan 13, 2020'); // Returns: '2020-01-13T00:00:00.000Z' ``` ``` -------------------------------- ### newUrl(params) Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Constructs a Google Trends search URL with query parameters. ```APIDOC ## newUrl(params) ### Description Constructs a Google Trends search URL with query parameters. ### Parameters - **params.geo** (string) - Optional - Geographic area code (e.g., 'US', 'GB') - **params.category** (string) - Optional - Category code (e.g., '5' for Computers & Electronics) - **params.searchTerm** (string) - Required - The search term to query - **params.timeRangeToUse** (string) - Optional - Time range code (e.g., 'now 1-d', 'today 1-m') ### Return Type string - A fully formed Google Trends URL with all query parameters encoded ### Example ```javascript const url = newUrl({ geo: 'US', category: '5', searchTerm: 'web scraping', timeRangeToUse: 'today 1-m' }); ``` ``` -------------------------------- ### Scrape from Google Sheet Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/README.md Input configuration for scraping search terms directly from a public Google Spreadsheet. ```json { "spreadsheetId": "1mZq8Z_q7K_r8Y_t7K_q1Z_w2X_y3Z_a4B_c5D_e6F_g7H", "isPublic": true, "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Define Output Dataset Structure Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Specifies the JSON schema for successful results, failed request debugging, and empty data handling. ```json Each item: { "Term / Date": "search term", // Column title from sheet or default "2019-01-13T00:00:00.000Z": "92", // ISO date if outputAsISODate: true "2019-01-20T00:00:00.000Z": "100", // or original format ... [customField]: customValue, // From extendOutputFunction ... } Failed requests: { "#debug": { url: "...", attempts: 3, errorMessages: ["..."], lastErrorType: "..." } } No data results: { "Term / Date": "term", "message": "The search term displays no data." } ``` -------------------------------- ### Error Handling Flow Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Visualizes the retry logic and error handling strategy for page loading, rate limits, and extraction failures. ```text Request starts ↓ Page loading → Timeout? → Retry (up to 3 times) ↓ 429 Rate Limit? → Sleep 10s → Retry ↓ Data extraction → Exception? → Log, push debug info ↓ Function execution → Error? → Log warning, use default output ↓ Final failure? → Push debug info, continue to next request ``` -------------------------------- ### URL Generation Logic Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Flow for generating search URLs and the resulting request options structure. ```javascript Search Terms OR Spreadsheet → { For each term: newUrl({ searchTerm, geo, category, timeRangeToUse (customTimeRange || timeRange) }) → URL with all parameters } → [RequestOptions] array with label: 'SEARCH' ``` ```javascript Input: { searchTerms: ['python'], geo: 'US', category: '5' } Output: [ { url: 'https://trends.google.com/trends/explore?q=python&geo=US&date=&cat=5', userData: { label: 'SEARCH' } } ] ``` -------------------------------- ### Scrape Specific Terms Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/README.md Input configuration for scraping trends for a list of specific search terms. ```json { "searchTerms": ["python", "javascript", "rust"], "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### checkAndEval Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Validates and evaluates the provided extend output function string. ```APIDOC ## checkAndEval ### Description Validates and evaluates the extend output function string to ensure it is a valid function. ### Signature `checkAndEval(extendOutputFunction: string | null | undefined): void` ### Parameters - **extendOutputFunction** (string | null | undefined) - Optional - JavaScript function code to evaluate ### Return Type void ### Throws - **Error** - Thrown if the function string is invalid JavaScript or the evaluated result is not a function. ### Example ```javascript checkAndEval('($) => { return { test: 123 }; }'); // Valid ``` ``` -------------------------------- ### Recent Time Ranges Configuration Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Configuration for relative time ranges like the past hour or past 7 days. ```json { "searchTerms": ["bitcoin"], "timeRange": "now 1-H", "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Request Queue Lifecycle Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Sequence of request management within the Apify crawler. ```javascript START Request → Added first to pre-warm Google Trends ↓ SEARCH Requests → Added in loop from sources array ↓ Request Queue → Managed by Apify crawler ``` -------------------------------- ### applyFunction(page, extendOutputFunction) Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/utils.md Executes a custom extend output function on the Puppeteer page context and merges the results. ```APIDOC ## applyFunction(page, extendOutputFunction) ### Description Executes the extend output function on the page context and merges results. It injects jQuery into the page and evaluates the provided function code. ### Parameters - **page** (Puppeteer.Page) - Required - Puppeteer page object - **extendOutputFunction** (string) - Optional - Custom output function code ### Return Type Promise - Empty object if no function provided, otherwise the return value from the evaluated function ### Example ```javascript const result = await applyFunction(page, '($) => { return { custom: "field" }; }'); ``` ``` -------------------------------- ### Combine Multiple Filters and Custom Output Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Applies geographic, category, and time filters while using an extension function to modify output fields. ```json { "searchTerms": [ "apartment", "house", "condo", "townhouse" ], "geo": "US", "category": "29", "customTimeRange": "2023-01-01 2023-12-31", "maxConcurrency": 5, "outputAsISODate": true, "extendOutputFunction": "($) => { return { property_type: 'residential' }; }", "proxyConfiguration": { "useApifyProxy": true } } ``` ```json { "Term / Date": "apartment", "2023-01-01T00:00:00.000Z": "85", "2023-01-08T00:00:00.000Z": "88", "property_type": "residential" } ``` -------------------------------- ### Define ProxyConfig Type Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/types.md Configuration object for proxy settings, supporting Apify Proxy or custom proxy URLs. ```javascript type ProxyConfig = { useApifyProxy?: boolean, proxyUrls?: string[], groups?: string[], newUrl?: () => string } ``` -------------------------------- ### Fix Invalid spreadsheetId Length Error Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/errors.md Verify that the spreadsheet ID string is exactly 44 characters long. ```text // Verify your Google Sheet ID is exactly 44 characters // In Google Sheets URL: // https://docs.google.com/spreadsheets/d/{THIS_IS_YOUR_ID}/edit // // Count characters carefully. Use a string length checker. // Example of correct format (44 characters): "1mZq8Z_q7K_r8Y_t7K_q1Z_w2X_y3Z_a4B_c5D_e6F_g7H" ``` -------------------------------- ### Tune Concurrency and Timeouts Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Adjusts parallel page processing and load timeouts for slower networks or rate-limiting mitigation. ```json { "searchTerms": ["term1", "term2", "term3"], "maxConcurrency": 3, "pageLoadTimeoutSecs": 240, "proxyConfiguration": { "useApifyProxy": true } } ``` -------------------------------- ### Fix Invalid maxItems Type Error Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/errors.md Ensure maxItems is provided as a number. ```json // Incorrect: { "maxItems": "100" } // Correct: { "maxItems": 100 } ``` -------------------------------- ### Configure Spreadsheet ID Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/configuration.md Specify the Google Sheet ID for importing search terms. ```json { "spreadsheetId": "1mZq8Z_q7K_r8Y_t7K_q1Z_w2X_y3Z_a4B_c5D_e6F_g7H" } ``` -------------------------------- ### Validate Input Logic Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Logic flow for validating input parameters in src/utils.js. ```javascript Input → Validate → { ✓ INPUT exists ✓ searchTerms OR spreadsheetId provided ✓ Field types correct ✓ SpreadsheetId format valid (44 chars) ✗ Throw Error } ``` -------------------------------- ### Export BASE_URL constant Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/api-reference/constants.md The module export definition for the BASE_URL constant. ```javascript exports.BASE_URL = 'https://trends.google.com/trends/explore'; ``` -------------------------------- ### Predefined Time Range Values Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/INDEX.md Use these string values to specify the time range for Google Trends data retrieval. ```text "" (empty) = Past 12 months "now 1-H" = Past hour "now 4-H" = Past 4 hours "now 1-d" = Past day "now 7-d" = Past 7 days "today 1-m" = Past 30 days "today 3-m" = Past 90 days "today 5-y" = Past 5 years "all" = 2004 - present ``` -------------------------------- ### Crawler Completion Logic Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/data-flow.md Describes the final state of the crawler once processing is finished or limits are met. ```text All requests processed OR maxItems reached ↓ Crawler stops ↓ Dataset contains all results ↓ Ready for download / export ``` -------------------------------- ### Invoke Google Trends Scraper via Apify SDK Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/usage-examples.md Use the Apify SDK to call the actor programmatically with specific search terms, time ranges, and proxy configurations. ```javascript const Apify = require('apify'); // Call the actor from another actor const run = await Apify.call('emastra/actor-google-trends-scraper', { searchTerms: ['python', 'javascript'], timeRange: 'today 1-m', geo: 'US', proxyConfiguration: { useApifyProxy: true } }); // Get results const results = run.output.body; console.log(results); ``` -------------------------------- ### Define CrawlerInput Type Source: https://github.com/emastra/actor-google-trends-scraper/blob/master/_autodocs/types.md Represents the complete input configuration for the actor. Either searchTerms or spreadsheetId must be provided. ```javascript type CrawlerInput = { searchTerms: string[], spreadsheetId?: string, isPublic?: boolean, timeRange?: string, category?: string, maxItems?: number, customTimeRange?: string | null, geo?: string, extendOutputFunction?: string | null, proxyConfiguration?: ProxyConfig, outputAsISODate?: boolean, maxConcurrency?: number, pageLoadTimeoutSecs?: number } ```