### Basic Scraper Initialization Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Example of creating a scraper with essential options: company ID, start date, and enabling browser visibility. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; import puppeteer from 'puppeteer'; // Example 1: Basic configuration const scraper1 = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01'), showBrowser: true }); ``` -------------------------------- ### Install israeli-bank-scrapers Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/README.md Install the package using npm. This is the first step to using the library. ```sh npm install israeli-bank-scrapers --save ``` -------------------------------- ### Multi-User Parallel Scraping Setup Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/configuration.md This example demonstrates how to configure and run scrapers for multiple users in parallel using separate browser contexts. It requires launching a Puppeteer browser and managing user credentials. ```typescript import puppeteer from 'puppeteer'; const browser = await puppeteer.launch({ args: ['--no-sandbox', '--disable-dev-shm-usage'] }); const users = [ { username: 'user1', password: 'pass1' }, { username: 'user2', password: 'pass2' }, { username: 'user3', password: 'pass3' } ]; const results = await Promise.all( users.map(async (credentials) => { const context = await browser.createBrowserContext(); const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01'), browserContext: context }); const result = await scraper.scrape(credentials); await context.close(); return result; }) ); await browser.close(); ``` -------------------------------- ### Install israeli-bank-scrapers-core Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/README.md Use this command to install the core version of the library, which is useful for applications that bundle node_modules like Electron applications. ```sh npm install israeli-bank-scrapers-core --save ``` -------------------------------- ### Comprehensive Progress Tracking Example Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/scraper-metadata.md A detailed example demonstrating how to track and log the duration of each stage in the scraper's lifecycle using the `onProgress` event handler. It logs messages for each stage and provides a final timeline summary. ```typescript import { CompanyTypes, createScraper, ScraperProgressTypes } from 'israeli-bank-scrapers'; const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01') }); // Create progress tracker const progressTracker = { startTime: Date.now(), stages: {} as Record }; scraper.onProgress((companyId, event) => { const elapsed = Date.now() - progressTracker.startTime; progressTracker.stages[event.type] = elapsed; switch (event.type) { case ScraperProgressTypes.Initializing: console.log('โณ Initializing browser...'); break; case ScraperProgressTypes.StartScraping: console.log('๐Ÿ” Starting scrape...'); break; case ScraperProgressTypes.LoggingIn: console.log('๐Ÿ” Authenticating...'); break; case ScraperProgressTypes.LoginSuccess: console.log('โœ… Login successful'); console.log(` Time: ${elapsed}ms`); break; case ScraperProgressTypes.LoginFailed: console.log('โŒ Login failed'); break; case ScraperProgressTypes.ChangePassword: console.log('โš ๏ธ Password change required'); break; case ScraperProgressTypes.EndScraping: console.log('โœ”๏ธ Scraping completed'); break; case ScraperProgressTypes.Terminating: console.log('๐Ÿงน Cleaning up...'); console.log(` Total time: ${elapsed}ms`); break; } }); // Perform scraping const result = await scraper.scrape({ username: 'user123', password: 'pass456' }); // Output progress summary console.log(' === Progress Timeline ==='); Object.entries(progressTracker.stages).forEach(([stage, time]) => { console.log(`${stage}: ${time}ms`); }); ``` -------------------------------- ### Scraper Initialization with Opt-In Features Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Demonstrates enabling specific opt-in features for a scraper. This example enables a Mizrahi-specific feature for handling pending transactions. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; import puppeteer from 'puppeteer'; // Example 6: With opt-in features const scraper6 = createScraper({ companyId: CompanyTypes.mizrahi, startDate: new Date('2024-01-01'), optInFeatures: ['mizrahi:pendingIfNoIdentifier'] }); ``` -------------------------------- ### Usage Example for Constants Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/api-reference.md Demonstrates how to import and use currency and date format constants for formatting and validating dates, and displaying amounts with currency symbols. ```typescript import { SHEKEL_CURRENCY, ISO_DATE_FORMAT, ISO_DATE_REGEX } from 'israeli-bank-scrapers'; import moment from 'moment'; // Format a date as ISO string const txnDate = moment().format(ISO_DATE_FORMAT); // Validate an ISO date string const isValidIsoDate = ISO_DATE_REGEX.test('2024-01-15T10:30:45.123Z'); // Display amounts with proper currency function formatAmount(amount: number, currency: string) { if (currency === SHEKEL_CURRENCY) { return `โ‚ช${amount.toFixed(2)}`; } return `${amount.toFixed(2)} ${currency}`; } ``` -------------------------------- ### Example Usage of OutputDataOptions Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Demonstrates how to use OutputDataOptions to include transactions before the specified startDate. This is useful when you need a complete transaction history. ```typescript const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01'), outputData: { enableTransactionsFilterByDate: true // Include transactions before startDate } }); const result = await scraper.scrape(credentials); // result.accounts[].txns will include transactions from before 2024-01-01 ``` -------------------------------- ### Scraper Initialization with Page Customization Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Shows how to provide a `preparePage` callback to modify the browser page before scraping begins. This example sets a custom user agent. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; import puppeteer from 'puppeteer'; // Example 4: With page customization const scraper4 = createScraper({ companyId: CompanyTypes.israelcard, startDate: new Date('2024-01-01'), preparePage: async (page) => { await page.setUserAgent('Custom User Agent'); } }); ``` -------------------------------- ### Scraping and Displaying Transactions Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Example of how to use the scraper to fetch account data and iterate through transactions, displaying key details for each. ```typescript const result = await scraper.scrape(credentials); if (result.success) { result.accounts.forEach(account => { account.txns.forEach(txn => { const amountDisplay = `${txn.chargedAmount} ${txn.chargedCurrency || txn.originalCurrency}`; console.log(`[${txn.status.toUpperCase()}] ${txn.date}`); console.log(` ${txn.description}`); console.log(` Amount: ${amountDisplay}`); if (txn.type === 'installments' && txn.installments) { console.log(` Installment: ${txn.installments.number}/${txn.installments.total}`); } if (txn.memo) { console.log(` Memo: ${txn.memo}`); } if (txn.category) { console.log(` Category: ${txn.category}`); } }); }); } ``` -------------------------------- ### Scraper Usage Examples Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Demonstrates how to create and use scrapers for different banks (Leumi, Discount, Isracard) with their respective credential formats. Ensure you import the necessary functions and types. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; // Leumi Bank const leumiScraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date() }); const leumiResult = await leumiScraper.scrape({ username: 'username123', password: 'password456' }); // Discount Bank const discountScraper = createScraper({ companyId: CompanyTypes.discount, startDate: new Date() }); const discountResult = await discountScraper.scrape({ id: '123456789', password: 'password456', num: '9876' }); // Isracard const isracardScraper = createScraper({ companyId: CompanyTypes.isracard, startDate: new Date() }); const isracardResult = await isracardScraper.scrape({ id: '123456789', card6Digits: '123456', password: 'password456' }); ``` -------------------------------- ### CI/CD Test Configuration Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/CONTRIBUTING.md Example JSON configuration for providing test credentials and options via the TESTS_CONFIG environment variable for CI/CD services. ```json { "options": { "startDate": "2019-06-01", "combineInstallments": false, "showBrowser": true, "verbose": false, "args": [] }, "credentials": { "leumi": { "username": "demouser", "password": "demopassword" } }, "companyAPI": { "enabled": true, "invalidPassword": false } } ``` -------------------------------- ### Combine Installment Transactions Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/configuration.md Set `combineInstallments` to `true` to consolidate installment transactions into a single entry, rather than having separate entries for each month. When enabled, only the initial installment transaction with the full amount is included. ```typescript // Get combined installment view const scraper = createScraper({ companyId: CompanyTypes.isracard, startDate: new Date('2024-01-01'), combineInstallments: true }); const result = await scraper.scrape(credentials); // result.accounts[].txns only contains initial installment transactions ``` -------------------------------- ### Example Usage of FutureDebit Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Shows how to iterate through future debits returned by the scraper and log their details. This helps in tracking upcoming charges. ```typescript const result = await scraper.scrape(credentials); if (result.futureDebits) { result.futureDebits.forEach(debit => { console.log(`Scheduled: ${debit.amount} ${debit.amountCurrency} on ${debit.chargeDate}`); if (debit.bankAccountNumber) { console.log(` Account: ${debit.bankAccountNumber}`); } }); } ``` -------------------------------- ### INVALID_PASSWORD Error Example Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Handles cases where incorrect credentials are provided. Use this to prompt the user to re-enter their username and password. ```typescript { success: false, errorType: 'INVALID_PASSWORD', errorMessage: 'Invalid username or password' } ``` ```typescript const result = await scraper.scrape(credentials); if (!result.success && result.errorType === 'INVALID_PASSWORD') { console.error('Please verify your username and password'); // Prompt user to re-enter credentials } ``` -------------------------------- ### Minimal Bank Scraping Example Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/README.md Demonstrates how to create a scraper instance for a specific bank and scrape account transactions. Ensure you have the correct company ID and date range. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01') }); const result = await scraper.scrape({ username: 'username123', password: 'password456' }); if (result.success) { console.log(result.accounts); } else { console.error(result.errorMessage); } ``` -------------------------------- ### Usage Example: Accessing and Displaying Scraper Metadata Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/scraper-metadata.md Demonstrates how to import scraper metadata, access specific bank details, and iterate through all supported banks to display their information. ```typescript import { SCRAPERS, CompanyTypes } from 'israeli-bank-scrapers'; // Get metadata for a specific bank const leumiMeta = SCRAPERS[CompanyTypes.leumi]; console.log(leumiMeta.name); // "Bank Leumi" console.log(leumiMeta.loginFields); // ["username", "password"] // Display all supported banks Object.entries(SCRAPERS).forEach(([companyId, meta]) => { console.log(`${companyId}: ${meta.name}`); console.log(` Required fields: ${meta.loginFields.join(', ')}`); }); // Build dynamic login form based on scraper requirements function buildLoginForm(companyId: CompanyTypes) { const meta = SCRAPERS[companyId]; return meta.loginFields.map(field => ({ name: field, label: field.charAt(0).toUpperCase() + field.slice(1), type: field === 'password' ? 'password' : 'text' })); } // Generate HTML form const form = buildLoginForm(CompanyTypes.isracard); // Output: [ // { name: 'id', label: 'Id', type: 'text' }, // { name: 'card6Digits', label: 'Card6Digits', type: 'text' }, // { name: 'password', label: 'Password', type: 'password' } // ] ``` -------------------------------- ### runSerial Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Executes an array of asynchronous functions sequentially, ensuring each completes before the next one starts. ```APIDOC ## runSerial ### Description Execute an array of async functions sequentially. Each function in the array is invoked, and its promise must resolve before the next function is called. ### Signature ```typescript function runSerial(actions: (() => Promise)[]): Promise ``` ### Parameters #### Path Parameters * None #### Query Parameters * None #### Request Body * None ### Parameters * **actions** (Array) - Required - Array of functions returning promises. ### Returns An array containing the results of each executed function, in the order they were executed. ### Example ```typescript // Fetch transactions from multiple months in order const results = await runSerial([ () => scraper.fetchMonth('2024-01'), () => scraper.fetchMonth('2024-02'), () => scraper.fetchMonth('2024-03') ]); console.log('Fetched', results.length, 'months'); ``` ``` -------------------------------- ### Minimum Required Configuration Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/README.md This snippet shows the essential settings needed to initialize a bank scraper. Specify the company ID for the desired bank and the start date for data retrieval. ```typescript { companyId: CompanyTypes.leumi, // Which bank startDate: new Date('2024-01-01') // From when } ``` -------------------------------- ### Usage Examples for Constants Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/scraper-metadata.md Demonstrates how to use the currency and date constants for formatting currency amounts, converting dates to ISO strings, and validating date strings. Requires the 'moment' library. ```typescript import moment from 'moment'; import { SHEKEL_CURRENCY, DOLLAR_CURRENCY, ISO_DATE_FORMAT, ISO_DATE_REGEX } from 'israeli-bank-scrapers'; // Format amount with currency function formatCurrency(amount: number, currency: string): string { if (currency === SHEKEL_CURRENCY) { return `${SHEKEL_CURRENCY_SYMBOL}${amount.toFixed(2)}`; } else if (currency === DOLLAR_CURRENCY) { return `${DOLLAR_CURRENCY_SYMBOL}${amount.toFixed(2)}`; } return `${amount.toFixed(2)} ${currency}`; } // Format date in ISO format function toIsoString(date: Date): string { return moment(date).format(ISO_DATE_FORMAT); } // Validate ISO date string function isValidIsoDate(dateStr: string): boolean { return ISO_DATE_REGEX.test(dateStr); } // Example usage console.log(formatCurrency(1500.50, SHEKEL_CURRENCY)); // โ‚ช1500.50 console.log(formatCurrency(50.00, DOLLAR_CURRENCY)); // $50.00 console.log(toIsoString(new Date('2024-01-15'))); // 2024-01-15T... console.log(isValidIsoDate('2024-01-15T10:30:45.123Z')); // true ``` -------------------------------- ### ACCOUNT_BLOCKED Error Example Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Handles cases where the user's account is locked by the bank. Advise the user to contact the bank for resolution. ```typescript { success: false, errorType: 'ACCOUNT_BLOCKED', errorMessage: 'Your account has been temporarily blocked' } ``` ```typescript const result = await scraper.scrape(credentials); if (!result.success && result.errorType === 'ACCOUNT_BLOCKED') { console.error('Account is blocked. Please contact the bank.'); // Suggest user contact support showAlert('Please contact your bank to unblock your account'); } ``` -------------------------------- ### Full Scraper Configuration with Options Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/configuration.md This example shows a comprehensive configuration including bank selection, date range, browser settings, behavior options, page customization, output settings, and feature flags. It requires importing necessary types and launching a Puppeteer browser instance. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; import puppeteer from 'puppeteer'; const browser = await puppeteer.launch(); const scraper = createScraper({ // Bank selection companyId: CompanyTypes.mizrahi, // Date range startDate: new Date('2024-01-01'), futureMonthsToScrape: 3, // Browser configuration browser, skipCloseBrowser: true, // Behavior options verbose: true, combineInstallments: false, additionalTransactionInformation: true, includeRawTransaction: true, // Page customization defaultTimeout: 60000, navigationRetryCount: 3, viewportSize: { width: 1920, height: 1080 }, storeFailureScreenShotPath: '/tmp/debug', preparePage: async (page) => { await page.setUserAgent('Custom User Agent'); }, // Output options outputData: { enableTransactionsFilterByDate: true }, // Feature flags optInFeatures: ['mizrahi:pendingIfNoIdentifier'] }); const result = await scraper.scrape({ username: 'user123', password: 'pass456' }); await browser.close(); ``` -------------------------------- ### Handle INITIALIZING Event Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/scraper-metadata.md Listen for the INITIALIZING event to perform actions before the scraper begins its setup. This is the first event emitted in the scraping lifecycle. ```typescript scraper.onProgress((companyId, event) => { if (event.type === 'INITIALIZING') { console.log('Setting up browser environment...'); } }); ``` -------------------------------- ### Fetch Last 3 Months of Transactions Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/configuration.md This example demonstrates how to set the startDate to fetch transactions from the last three months. Note that banks have varying limits on how far back data can be retrieved. ```typescript // Fetch last 3 months const threeMonthsAgo = new Date(); threeMonthsAgo.setMonth(threeMonthsAgo.getMonth() - 3); const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: threeMonthsAgo }); ``` -------------------------------- ### CLI Test Configuration via Environment Variable Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/CONTRIBUTING.md Example of setting the TESTS_CONFIG environment variable with a multiline JSON configuration using 'cat' for CLI execution on macOS. ```bash TESTS_CONFIG=`cat < { switch (event.type) { case 'INITIALIZING': console.log('Setting up browser...'); break; case 'LOGGING_IN': console.log('Logging in...'); break; case 'LOGIN_SUCCESS': console.log('Successfully logged in'); break; case 'LOGIN_FAILED': console.log('Login failed'); break; case 'CHANGE_PASSWORD': console.log('Server requires password change'); break; case 'END_SCRAPING': console.log('Scraping completed'); break; case 'TERMINATING': console.log('Cleaning up resources'); break; } }); await scraper.scrape(credentials); ``` -------------------------------- ### CHANGE_PASSWORD Error Example Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Handles situations where the bank requires a password change. Redirect the user to the bank's password change page. ```typescript { success: false, errorType: 'CHANGE_PASSWORD', errorMessage: 'Please change your password and try again' } ``` ```typescript const result = await scraper.scrape(credentials); if (!result.success && result.errorType === 'CHANGE_PASSWORD') { console.error('You must change your password to continue'); // Redirect user to bank's password change page openBrowser('https://bank.example.com/change-password'); } ``` -------------------------------- ### Create Scraper with Company ID and Start Date Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/configuration.md Use this snippet to initialize a scraper for a specific financial institution and set the earliest date for transaction fetching. The companyId determines the scraper implementation, and startDate is constrained by bank-specific limits. ```typescript const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01') }); ``` -------------------------------- ### OneZeroScraper Usage Example with OTP Callback Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/api-reference.md Demonstrates how to use the OneZeroScraper with an OTP callback function to handle two-factor authentication. This is useful for interactive sessions where the user can provide the OTP code. ```typescript import { CompanyTypes, createScraper, OneZeroScraper } from 'israeli-bank-scrapers'; // Option 1: Using OTP callback const scraper = createScraper({ companyId: CompanyTypes.oneZero, startDate: new Date('2024-01-01') }); const result = await scraper.scrape({ email: 'user@example.com', password: 'mypassword', phoneNumber: '0501234567', otpCodeRetriever: async () => { // Prompt user for OTP code return await getUserOtpInput(); } }); ``` -------------------------------- ### Example GENERAL_ERROR Response Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Illustrates the structure of a GENERAL_ERROR response, indicating a system or browser-level failure. This error typically occurs during browser initialization or cleanup. ```typescript { success: false, errorType: 'GENERAL_ERROR', errorMessage: 'Failed to launch browser' } ``` -------------------------------- ### Get Scraper Metadata Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/README.md Import and use the SCRAPERS list to retrieve metadata for available scrapers, including their names and required login fields. ```javascript import { SCRAPERS } from 'israeli-bank-scrapers'; ``` -------------------------------- ### Common Optional Configuration Settings Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/README.md These are frequently used optional parameters to customize scraper behavior. Adjust settings like browser visibility, installment handling, and timeouts as needed. ```typescript { showBrowser: true, // See browser (debugging) combineInstallments: false, // Combine installment txns additionalTransactionInformation: false, // Fetch categories navigationRetryCount: 3, // Retry failed nav defaultTimeout: 60000, // Page timeout optInFeatures: [] // Opt-in breaking changes } ``` -------------------------------- ### Processing Scraped Account Data Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Demonstrates how to process the results of a scraper operation, iterating through accounts to display balance and transaction summaries. This example assumes the scraper operation was successful. ```typescript const result = await scraper.scrape(credentials); if (result.success) { result.accounts.forEach(account => { console.log(`Account ${account.accountNumber}:`); if (account.balance !== undefined) { console.log(` Current Balance: ${account.balance}`); } const total = account.txns.reduce((sum, t) => sum + t.chargedAmount, 0); console.log(` Total Charged: ${total}`); console.log(` Transactions: ${account.txns.length}`); }); } ``` -------------------------------- ### Example TWO_FACTOR_RETRIEVER_MISSING Response Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Shows the response format when a scraper requires a two-factor authentication callback but none is provided. This error occurs when using scrapers like OneZero without the necessary authentication details. ```typescript { success: false, errorType: 'TWO_FACTOR_RETRIEVER_MISSING', errorMessage: 'otpCodeRetriever is required for OneZero scraper' } ``` -------------------------------- ### OneZeroScraper Usage Example with Long-Term Token Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/api-reference.md Shows how to use the OneZeroScraper with a pre-generated long-term token for authentication. This is suitable for non-interactive or automated scenarios where the OTP code is not readily available. ```typescript // Option 2: Using long-term token const resultWithToken = await scraper.scrape({ email: 'user@example.com', password: 'mypassword', otpLongTermToken: 'eyJraWQiOiJiNzU3OGM5Yy0wM2YyLTRkMzktYjBm...' }); ``` -------------------------------- ### Enable Opt-In Features for Mizrahi Bank Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/configuration.md Use the `optInFeatures` array to enable experimental or breaking-change features. This example shows enabling pending transaction flags for Mizrahi bank. ```typescript const scraper = createScraper({ companyId: CompanyTypes.mizrahi, startDate: new Date('2024-01-01'), optInFeatures: [ 'mizrahi:pendingIfNoIdentifier', 'mizrahi:pendingIfTodayTransaction' ] }); ``` -------------------------------- ### Implement Scheduled Scraping Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/integration-patterns.md Use this class to schedule periodic scraping of bank data. It runs the scrape task immediately on start and then according to the provided cron schedule. Configure banks, schedule, and callbacks for success and error. ```typescript import cron from 'node-cron'; import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; class ScheduledScraper { constructor( private config: { banks: Array<{ companyId: CompanyTypes; credentials: any; }>; schedule: string; // Cron expression onSuccess?: (results: any) => void; onError?: (error: Error) => void; } ) {} start() { // Run on schedule cron.schedule(this.config.schedule, () => { this.scrapeAll().catch(this.config.onError); }); // Also run immediately this.scrapeAll().catch(this.config.onError); console.log(`Scraper scheduled: ${this.config.schedule}`); } private async scrapeAll() { console.log(`[${new Date().toISOString()}] Starting scheduled scrape`); const results = await Promise.all( this.config.banks.map(async (bank) => { try { const scraper = createScraper({ companyId: bank.companyId, startDate: new Date('2024-01-01') }); const result = await scraper.scrape(bank.credentials); return { companyId: bank.companyId, success: result.success, accounts: result.success ? result.accounts : [], timestamp: new Date().toISOString() }; } catch (error) { return { companyId: bank.companyId, success: false, error: (error as Error).message, timestamp: new Date().toISOString() }; } }) ); if (this.config.onSuccess) { this.config.onSuccess(results); } console.log(`Scrape completed with ${results.filter(r => r.success).length}/${results.length} success`); } } // Usage const scheduler = new ScheduledScraper({ banks: [ { companyId: CompanyTypes.leumi, credentials: { username: 'u1', password: 'p1' } }, { companyId: CompanyTypes.hapoalim, credentials: { userCode: 'c1', password: 'p2' } } ], schedule: '0 2 * * *', // 2 AM daily onSuccess: (results) => { // Save to database or send notification console.log('Scrape results:', results); }, onError: (error) => { // Log or alert console.error('Scraping error:', error); } }); scheduler.start(); ``` -------------------------------- ### Scraping Result Usage Example Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Demonstrates how to handle the ScraperScrapingResult object. It checks for success and logs account details or error messages accordingly. Supports handling of future debits. ```typescript const result = await scraper.scrape(credentials); if (result.success) { console.log(`Scraping succeeded for ${result.accounts.length} accounts`); result.accounts.forEach(account => { console.log(`Account: ${account.accountNumber}`); console.log(` Balance: ${account.balance}`); console.log(` Transactions: ${account.txns.length}`); account.txns.forEach(txn => { console.log(` ${txn.date}: ${txn.chargedAmount} ${txn.chargedCurrency || 'ILS'}`); console.log(` ${txn.description}`); }); }); if (result.futureDebits && result.futureDebits.length > 0) { console.log('Upcoming scheduled charges:'); result.futureDebits.forEach(charge => { console.log(` ${charge.chargeDate}: ${charge.amount} ${charge.amountCurrency}`); }); } } else { console.error(`Scraping failed: ${result.errorType}`); console.error(` Message: ${result.errorMessage}`); } ``` -------------------------------- ### filterOldTransactions Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Filters out transactions that are older than a specified start date. Optionally combines installments, returning only the initial ones if true. ```APIDOC ## filterOldTransactions ### Description Filter out transactions older than a start date. ### Signature ```typescript function filterOldTransactions( txns: Transaction[], startMoment: Moment, combineInstallments: boolean ): Transaction[]; ``` ### Parameters #### Path Parameters - **txns** (Transaction[]) - Yes - Transactions to filter - **startMoment** (Moment) - Yes - Cutoff date (moment object) - **combineInstallments** (boolean) - Yes - If true, only return initial installments ### Returns Filtered transaction array ### Request Example ```typescript import moment from 'moment'; const cutoff = moment('2024-01-01'); const recent = filterOldTransactions( account.txns, cutoff, false ); ``` ``` -------------------------------- ### Filter Old Transactions Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Filters out transactions older than a specified start date. Can optionally combine installments to only include the first instance. ```typescript import moment from 'moment'; const cutoff = moment('2024-01-01'); const recent = filterOldTransactions( account.txns, cutoff, false ); ``` -------------------------------- ### Get Puppeteer Configuration Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/api-reference.md Retrieves the required Chromium revision number for Puppeteer. Use this when managing your own Chromium installation, especially in bundled applications like Electron. ```typescript import { getPuppeteerConfig } from 'israeli-bank-scrapers-core'; const config = getPuppeteerConfig(); console.log(`Required Chromium revision: ${config.chromiumRevision}`); // Output: Required Chromium revision: 1250580 // Use this revision number with download-chromium or similar tools // to fetch the correct Chromium version for your environment ``` -------------------------------- ### Transaction Installments Interface Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Defines the structure for installment details when a transaction is part of an installment plan. ```typescript interface TransactionInstallments { number: number; // Current installment number total: number; // Total number of installments } ``` -------------------------------- ### fixInstallments Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Adjusts the dates of installment transactions to accurately reflect when each installment was charged. This is crucial for accurate financial tracking of recurring payments. ```APIDOC ## fixInstallments ### Description Adjust dates of installment transactions to reflect actual charge dates. For installment transactions where installment.number > 1, adds months to the date to represent when that installment was actually charged. ### Signature ```typescript function fixInstallments(txns: Transaction[]): Transaction[]; ``` ### Returns Array of transactions with adjusted installment dates. ### Request Example ```typescript // Input: Installment 2 of 3 dated 2024-01-15 // Output: Date adjusted to 2024-02-15 (one month later) const fixed = fixInstallments(transactions); ``` ``` -------------------------------- ### Fix Installment Transaction Dates Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Adjusts the dates of installment transactions to reflect their actual charge dates. Use when dealing with multi-part payments to ensure accurate chronological ordering. ```typescript // Input: Installment 2 of 3 dated 2024-01-15 // Output: Date adjusted to 2024-02-15 (one month later) const fixed = fixInstallments(transactions); ``` -------------------------------- ### Transaction Type and Status Enums Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Defines the possible types for a transaction ('normal' or 'installments') and its status ('completed' or 'pending'). ```typescript enum TransactionTypes { Normal = 'normal', // Regular single transaction Installments = 'installments' // Part of installment plan } enum TransactionStatuses { Completed = 'completed', // Transaction has cleared Pending = 'pending' // Transaction is pending } ``` -------------------------------- ### GENERIC Error Example Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Handles unhandled exceptions during scraping. Log the detailed error message for debugging purposes. ```typescript { success: false, errorType: 'GENERIC', errorMessage: 'Failed to load transactions page' } ``` ```typescript const result = await scraper.scrape(credentials); if (!result.success && result.errorType === 'GENERIC') { console.error(`Generic error occurred: ${result.errorMessage}`); // Log detailed error for debugging logger.error('Generic scraping error', { companyId: options.companyId, message: result.errorMessage }); } ``` -------------------------------- ### Run All Tests Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/CONTRIBUTING.md Execute all tests for companies for which credentials have been provided in the configuration file. ```bash npm test ``` -------------------------------- ### getAllMonthMoments Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Get an array of moment objects for each month in a date range. This function is useful for iterating through months for reporting or analysis. ```APIDOC ## getAllMonthMoments ### Description Get an array of moment objects for each month in a date range. ### Signature ```typescript function getAllMonthMoments( startMoment: Moment | string, futureMonths?: number ): Moment[]; ``` ### Parameters #### Path Parameters - **startMoment** (Moment | string) - Yes - Start date (moment or string) - **futureMonths** (number) - No - Number of months into future to include ### Returns Array of moment objects, one per month from start to current/future month ### Request Example ```typescript import getAllMonthMoments from 'israeli-bank-scrapers/helpers'; import moment from 'moment'; // Get months from January to now const months = getAllMonthMoments('2024-01-01'); console.log(months.length); // ~12 (depending on current date) // Get months from January to 3 months in future const futureMonths = getAllMonthMoments('2024-01-01', 3); console.log(futureMonths.length); // ~15 // Use with moment const start = moment('2024-01-15'); const allMonths = getAllMonthMoments(start); allMonths.forEach(month => { console.log(month.format('YYYY-MM')); }); ``` ``` -------------------------------- ### Get Current Page URL (Server-Side) Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Retrieves the current URL of the page as determined by the server. This is the default and synchronous method. ```typescript // Get server-side URL const url = getCurrentUrl(page); console.log('Current URL:', url); ``` -------------------------------- ### Customize Browser Instance with prepareBrowser Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/configuration.md Utilize the `prepareBrowser` async callback to customize the browser instance before scraping begins. This allows for adding event listeners or checking browser details. ```typescript const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01'), prepareBrowser: async (browser) => { // Add event listeners browser.on('error', err => console.error('Browser error:', err)); // Check browser version const version = await browser.version(); console.log('Browser version:', version); } }); ``` -------------------------------- ### Run Specific Test Suite Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/CONTRIBUTING.md Execute all tests within a specific 'describe' block (suite) using the 'testNamePattern' argument. ```bash npm test -- --testNamePattern="Leumi legacy scraper" ``` -------------------------------- ### TIMEOUT Error Example and Mitigation Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Handles operations exceeding the time limit. Consider increasing the default timeout or retry count for the scraper. ```typescript { success: false, errorType: 'TIMEOUT', errorMessage: 'Waiting for navigation timed out after 30000 ms' } ``` ```typescript const result = await scraper.scrape(credentials); if (!result.success && result.errorType === 'TIMEOUT') { console.error('Operation took too long. This might be a network issue.'); // Retry with increased timeout const retryOptions = { ...options, defaultTimeout: 60000 // Double the timeout }; const retryScraper = createScraper(retryOptions); } ``` ```typescript const scraper = createScraper({ companyId: CompanyTypes.leumi, startDate: new Date('2024-01-01'), defaultTimeout: 60000, // Increase timeout to 60 seconds navigationRetryCount: 3, // Retry failed navigation timeout: 45000 // Browser launch timeout }); ``` -------------------------------- ### Run Specific Test Case Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/CONTRIBUTING.md Execute a particular test case within a specific 'describe' block using the 'testNamePattern' argument. ```bash npm test -- --testNamePattern="Leumi legacy scraper should expose login fields in scrapers constant" ``` -------------------------------- ### BaseScraperWithBrowser.navigateTo Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/api-reference.md Navigates to a specified URL, with options for controlling when navigation is considered complete and how many times to retry on failure. ```APIDOC ## navigateTo ### Description Navigate to a URL with optional retry logic for network failures. ### Method `navigateTo` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters #### url (string) - Required - Target URL to navigate to #### waitUntil (PuppeteerLifeCycleEvent) - Optional - DOM event to wait for ('load', 'domcontentloaded', etc.) #### retries (number) - Optional - Number of times to retry on failure ### Request Example ```javascript // Example usage: await scraper.navigateTo('https://example.com', 'networkidle0', 3); ``` ### Response #### Success Response None (This method returns void) #### Response Example None ``` -------------------------------- ### Catching TWO_FACTOR_RETRIEVER_MISSING Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/errors.md Provides an example of how to detect and handle the TWO_FACTOR_RETRIEVER_MISSING error. This indicates that the scraper needs an OTP code retriever or a long-term token to proceed. ```typescript const result = await scraper.scrape(credentials); if (!result.success && result.errorType === 'TWO_FACTOR_RETRIEVER_MISSING') { console.error('OTP code retriever is required for this bank'); // Provide either an otpCodeRetriever callback or otpLongTermToken } ``` -------------------------------- ### Get All Month Moments Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/helpers.md Generates an array of moment objects for each month within a specified date range. Useful for iterating through months for reporting or analysis. ```typescript import getAllMonthMoments from 'israeli-bank-scrapers/helpers'; import moment from 'moment'; // Get months from January to now const months = getAllMonthMoments('2024-01-01'); console.log(months.length); // ~12 (depending on current date) // Get months from January to 3 months in future const futureMonths = getAllMonthMoments('2024-01-01', 3); console.log(futureMonths.length); // ~15 // Use with moment const start = moment('2024-01-15'); const allMonths = getAllMonthMoments(start); allMonths.forEach(month => { console.log(month.format('YYYY-MM')); }); ``` -------------------------------- ### Scraper Initialization with External Browser Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Demonstrates creating a scraper using an externally launched Puppeteer browser instance. The `skipCloseBrowser` option prevents the scraper from closing the provided browser. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; import puppeteer from 'puppeteer'; // Example 2: With external browser const browser = await puppeteer.launch(); const scraper2 = createScraper({ companyId: CompanyTypes.hapoalim, startDate: new Date('2024-01-01'), browser, skipCloseBrowser: true }); ``` -------------------------------- ### Get Chromium Revision for puppeteer-core Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/README.md This code snippet retrieves the specific chromium revision required by the puppeteer-core library, which is necessary when using the israeli-bank-scrapers-core variation. ```javascript import { getPuppeteerConfig } from 'israeli-bank-scrapers-core'; const chromiumVersion = getPuppeteerConfig().chromiumRevision; ``` -------------------------------- ### Import Currency Constants Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/README.md Import predefined constants for currency codes and symbols from the 'israeli-bank-scrapers' library. These constants represent common Israeli currency representations. ```typescript import { SHEKEL_CURRENCY, // 'ILS' DOLLAR_CURRENCY, // 'USD' EURO_CURRENCY, // 'EUR' SHEKEL_CURRENCY_SYMBOL, // 'โ‚ช' SHEKEL_CURRENCY_KEYWORD // 'ืฉ"ื—' } from 'israeli-bank-scrapers'; ``` -------------------------------- ### Reuse Browser Instance for Multiple Scrapers Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/integration-patterns.md Launch a single browser instance and reuse it for scraping multiple banks. Set `skipCloseBrowser: true` for each scraper and close the browser only after all scraping tasks are complete. This pattern improves performance by avoiding repeated browser startup overhead. ```typescript import puppeteer from 'puppeteer'; import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; async function scrapeWithSharedBrowser() { // Launch browser once const browser = await puppeteer.launch({ headless: true, args: ['--no-sandbox', '--disable-dev-shm-usage'] }); try { // Scrape multiple banks with same browser const scrapers = [ { companyId: CompanyTypes.leumi, credentials: { username: 'user1', password: 'pass1' } }, { companyId: CompanyTypes.hapoalim, credentials: { userCode: 'code1', password: 'pass2' } } ]; const results = await Promise.all( scrapers.map(async (scraper) => { const instance = createScraper({ companyId: scraper.companyId, startDate: new Date('2024-01-01'), browser, skipCloseBrowser: true // Don't close browser after scraping }); return instance.scrape(scraper.credentials); }) ); return results; } finally { // Close browser once after all scraping is done await browser.close(); } } ``` -------------------------------- ### Successful Scraping Sequence Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/scraper-metadata.md Illustrates the expected order of progress events during a successful scraping operation, from initialization to termination. ```text INITIALIZING โ†’ START_SCRAPING โ†’ LOGGING_IN โ†’ LOGIN_SUCCESS โ†’ END_SCRAPING โ†’ TERMINATING ``` -------------------------------- ### OneZeroScraper Methods Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/api-reference.md Provides details on the methods available for the OneZeroScraper, including initialization, scraping, and two-factor authentication handling. ```APIDOC ## OneZeroScraper Concrete scraper implementation for One Zero financial platform with two-factor authentication support. ### Class Declaration ```typescript class OneZeroScraper extends BaseScraperWithBrowser ``` ### Constructor ```typescript constructor(options: ScraperOptions) ``` ### Methods #### scrape ```typescript async scrape(credentials: OneZeroCredentials): Promise ``` **Description**: Initiates the scraping process with the provided credentials. #### triggerTwoFactorAuth ```typescript async triggerTwoFactorAuth(phoneNumber: string): Promise ``` **Description**: Triggers the two-factor authentication process for a given phone number. #### getLongTermTwoFactorToken ```typescript async getLongTermTwoFactorToken(otpCode: string): Promise ``` **Description**: Retrieves a long-term two-factor authentication token using an OTP code. ### Credentials ```typescript { email: string; password: string; phoneNumber: string; otpCodeRetriever: () => Promise; // OR otpLongTermToken?: string; } ``` **Description**: Defines the structure for credentials required by the OneZeroScraper. It includes email, password, phone number, and either an OTP code retriever function or a long-term token. ``` -------------------------------- ### DefaultBrowserOptions Configuration Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Configures the scraper to launch and manage its own browser instance. Options include showing the browser UI, specifying the executable path, and setting command-line arguments. ```typescript { showBrowser?: boolean; // default: false executablePath?: string; // Path to Chromium executable args?: string[]; // Chromium command-line flags timeout?: number; // Browser launch timeout (0 = no timeout) prepareBrowser?: (browser: Browser) => Promise; } ``` -------------------------------- ### Scraper Initialization for Additional Transaction Info Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/types.md Configures a scraper to fetch and include additional transaction details and the raw transaction object. This may increase scraping time. ```typescript import { CompanyTypes, createScraper } from 'israeli-bank-scrapers'; import puppeteer from 'puppeteer'; // Example 5: Collecting additional transaction info const scraper5 = createScraper({ companyId: CompanyTypes.mizrahi, startDate: new Date('2024-01-01'), additionalTransactionInformation: true, includeRawTransaction: true }); ``` -------------------------------- ### Failed Login Sequence Source: https://github.com/eshaham/israeli-bank-scrapers/blob/master/_autodocs/scraper-metadata.md Shows the sequence of events when the login process fails, leading to termination. ```text INITIALIZING โ†’ START_SCRAPING โ†’ LOGGING_IN โ†’ LOGIN_FAILED โ†’ END_SCRAPING โ†’ TERMINATING ```