### Google JWT & API Setup Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-search-console.md Example of setting up Google JWT for authentication. ```typescript const googleAuth = new JWT({ email: credentials.client_email, key: credentials.private_key, scopes: ['https://www.googleapis.com/auth/webmasters.readonly'] }); ``` -------------------------------- ### Manual Pagination - Request examples Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md Examples of constructing URLs for manual pagination, showing different start, num, and page values. ```typescript // Request page 1 scrapeURL(keyword, settings, countries, { start: 0, num: 10, page: 1 }) // Request page 2 scrapeURL(keyword, settings, countries, { start: 10, num: 10, page: 2 }) // Request page 10 (last page = 100 results) scrapeURL(keyword, settings, countries, { start: 90, num: 10, page: 10 }) ``` -------------------------------- ### Console Logging Examples Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Examples of structured log messages with different severity levels. ```text [ERROR] Scraping Keyword: example [SUCCESS] Updating the Keyword: example [WARN] Corrupt JSON detected in settings.json [INFO] Scraper: serpapi, Position: 5 ``` -------------------------------- ### Example Queries Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/database-models.md Provides various examples of how to query and manipulate Keyword data using Sequelize. ```typescript import Keyword from './models/keyword'; import { Op } from 'sequelize'; // Find keywords for domain const keywords = await Keyword.findAll({ where: { domain: 'example-com' } }); // Find by position (updating) const updating = await Keyword.findAll({ where: { updating: true } }); // Find by ID const keyword = await Keyword.findOne({ where: { ID: 42 } }); // Update position await keyword.update({ position: 10, history: JSON.stringify({...oldHistory, '2025-05-21': 10}), lastUpdated: new Date().toJSON(), updating: false }); // Bulk delete await Keyword.destroy({ where: { domain: 'example-com' } }); // Find with complex filter const keywords = await Keyword.findAll({ where: { domain: 'example-com', [Op.or]: [ { sticky: true }, { position: { [Op.lte]: 5 } } ] } }); ``` -------------------------------- ### getKeywordsInsight() Returns Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-search-console.md Example of keyword-level insights returned by getKeywordsInsight. ```typescript [ { keyword: "example", clicks: 50, impressions: 200, ctr: 0.25, position: 8, keywords: 1 }, { keyword: "another", clicks: 30, impressions: 150, ctr: 0.2, position: 15 } ] ``` -------------------------------- ### getPagesInsight() Returns Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-search-console.md Example of page-level insights returned by getPagesInsight. ```typescript [ { page: "https://example.com/", clicks: 80, impressions: 400, ctr: 0.2, position: 9 }, { page: "https://example.com/blog/", clicks: 20, impressions: 100, ctr: 0.2, position: 12 } ] ``` -------------------------------- ### Environment Variables Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/configuration.md An example of a .env.local file showing various configuration settings, including authentication, scraper API keys, Search Console credentials, SMTP details, and development environment settings. Sensitive fields like passwords and API keys are shown. ```dotenv # .env.local # Auth USER=admin PASSWORD=mysecurepassword123 SECRET=my-super-secret-key-min-32-chars! # Scraper SCRAPER_TYPE=serpapi SERP_API_KEY=your-serpapi-key # Search Console (global) SEARCH_CONSOLE_CLIENT_EMAIL=service-account@project.iam.gserviceaccount.com SEARCH_CONSOLE_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n" # Notifications SMTP_SERVER=smtp.sendpulse.com SMTP_PORT=587 SMTP_USERNAME=username@example.com SMTP_PASSWORD=smtppassword NOTIFICATION_EMAIL=owner@example.com # Development NODE_ENV=development ``` -------------------------------- ### Development Setup Commands Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md Common npm commands for setting up and running the development environment. ```bash npm install npm run dev # Start dev server on :3000 npm run build # Build production npm start # Start production server npm run db:migrate # Run migrations ``` -------------------------------- ### Parallel Scraping with Promise.all Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md TypeScript example demonstrating parallel scraping of keywords using Promise.all. ```typescript Promise.all(keywords.map(k => scrapeKeywordFromGoogle(k))) ``` -------------------------------- ### checkSerchConsoleIntegration() Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-search-console.md Example of validating Search Console credentials for a domain. ```typescript import { checkSerchConsoleIntegration } from './utils/searchConsole'; const domain = { domain: 'example.com', search_console: JSON.stringify({ client_email: 'sa@project.iam.gserviceaccount.com', private_key: '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n' }) }; const validation = await checkSerchConsoleIntegration(domain); if (!validation.isValid) { console.error(`Invalid credentials: ${validation.error}`); } ``` -------------------------------- ### getSearchConsoleApiInfo() Returns Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-search-console.md Example of the Search Console credentials object returned by getSearchConsoleApiInfo. ```typescript { property_type: "domain", url: "https://example.com", client_email: "sa@project.iam.gserviceaccount.com", private_key: "-----BEGIN PRIVATE KEY-----\n..." } ``` -------------------------------- ### Error and Warning Log Examples Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/errors.md Examples of log messages indicating errors and warnings. ```log [ERROR] Deleting Domain: example.com Error: ... [ERROR] Scraping Keyword: example keyword Error: Connection timeout [ERROR] Getting Domain Insight: example.com Error: API error [WARN] Corrupt JSON detected in data/settings.json. Backing up to data/settings.json.1234567890.corrupt ``` -------------------------------- ### refreshAndUpdateKeywords() Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-refresh-scraping.md Example usage of the refreshAndUpdateKeywords function. ```typescript import Keyword from './database/models/keyword'; import { getAppSettings } from './pages/api/settings'; import refreshAndUpdateKeywords from './utils/refresh'; const keywordIds = [1, 2, 3]; const keywords = await Keyword.findAll({ where: { ID: keywordIds } }); const settings = await getAppSettings(); const updated = await refreshAndUpdateKeywords(keywords, settings); console.log(`Updated ${updated.length} keywords`); ``` -------------------------------- ### fetchDomainSCData() Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-search-console.md Example usage of fetchDomainSCData to retrieve and log Search Console data. ```typescript import { fetchDomainSCData, getSearchConsoleApiInfo } from './utils/searchConsole'; const domain = await Domain.findOne({ where: { domain: 'example.com' } }); const scAPI = await getSearchConsoleApiInfo(domain); if (scAPI.client_email && scAPI.private_key) { const scData = await fetchDomainSCData(domain, scAPI); console.log(`SC Data: ${scData.sevenDays.length} items in 7-day`); } ``` -------------------------------- ### Scraper Not Set Up Error Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/errors.md Example JSON response when the scraper has not been configured. ```json { "error": "Scraper has not been set up yet." } ``` -------------------------------- ### getCountryInsight() Returns Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-search-console.md Example of country-level insights returned by getCountryInsight. ```typescript [ { country: "US", clicks: 100, impressions: 500, ctr: 0.2, position: 10, countries: 1 }, { country: "GB", clicks: 50, impressions: 250, ctr: 0.2, position: 12 } ] ``` -------------------------------- ### Update Settings via API Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/configuration.md Example of updating global settings using the PUT /api/settings endpoint. ```bash curl -X PUT http://localhost:3000/api/settings \ -H "Content-Type: application/json" \ -d '{ "settings": { "scraper_type": "serpapi", "scaping_api": "YOUR_API_KEY", "notification_interval": "daily", "notification_email": "owner@example.com", "scrape_strategy": "smart" } }' ``` -------------------------------- ### GET /api/dbmigrate Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response indicating if database migrations are pending. ```json { "hasMigrations": true } ``` -------------------------------- ### GET /api/ideas Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for fetching keyword ideas. ```json { "data": { "keywords": [ { "uid": "...", "keyword": "idea keyword", "competition": "MEDIUM", "competitionIndex": 50, "avgMonthlySearches": 1200, "country": "US", "domain": "example-com", "added": 0, "updated": 0, "position": 0 } ], "updated": 1234567890, "keywords_count": 25 } } ``` -------------------------------- ### History Storage Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/database-models.md Illustrates the JSON format for storing historical SERP positions. ```json { "2025-05-20": 5, "2025-05-21": 6, "2025-05-22": 5 } ``` -------------------------------- ### URL Construction - Example (SerpAPI) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md An example implementation of the scrapeURL function for SerpAPI. ```typescript scrapeURL: (keyword, settings, countries, pagination) => { const countryCode = keyword.country; // "US" const start = pagination?.start || 0; const num = pagination?.num || 100; return `https://serpapi.com/search?q=${encodeURIComponent(keyword.keyword)}&gl=${countryCode}&num=${num}&api_key=${apiKey}`; } ``` -------------------------------- ### GET /api/refresh Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for getting live search results for a keyword. ```json { "searchResult": { "keyword": "example", "country": "US", "position": 5, "results": [ {"position": 1, "url": "...", "title": "..."}, {"position": 2, "url": "...", "title": "..."} ] }, "error": "" } ``` -------------------------------- ### Scraper Not Set Up Error Response Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/errors.md Example JSON response when the scraper has not been configured. ```json { "started": false, "error": "Scraper has not been set up yet." } ``` -------------------------------- ### Safe JSON Reading Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-helpers.md Example usage of the safeReadJSON function to load settings from a JSON file. ```typescript const settings = await safeReadJSON( `${process.cwd()}/data/settings.json`, defaultSettings ); ``` -------------------------------- ### Serial Scraping with Delay Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md TypeScript example for serial scraping with a configurable delay between requests. ```typescript for (const keyword of keywords) { await scrapeKeywordWithStrategy(keyword); await sleep(settings.scrape_delay); } ``` -------------------------------- ### Example Domain Queries Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/database-models.md Demonstrates common operations using the Domain model, such as finding, updating, and creating domain records. ```typescript import Domain from './models/domain'; // Find domain by name const domain = await Domain.findOne({ where: { domain: 'example.com' } }); // Get all domains with stats const allDomains = await Domain.findAll(); // Update domain settings await domain.update({ notification_interval: 'weekly', scrape_strategy: 'smart' }); // Create new domain const newDomain = await Domain.create({ domain: 'newsite.com', slug: 'newsite-com', lastUpdated: new Date().toJSON(), added: new Date().toJSON() }); ``` -------------------------------- ### scrapeKeywordWithStrategy Example Usage Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-refresh-scraping.md An example demonstrating how to use the scrapeKeywordWithStrategy function in TypeScript, including handling the result. ```typescript import { scrapeKeywordWithStrategy } from './utils/scraper'; const keyword: KeywordType = { ID: 1, keyword: 'example', device: 'desktop', country: 'US', domain: 'example-com', position: 0, // ... other fields }; const result = await scrapeKeywordWithStrategy(keyword, settings); if (result && !result.error) { console.log(`Position: ${result.position}`); console.log(`URL: ${result.url}`); console.log(`Results: ${result.result.length}`); } else { console.error(`Error: ${result.error}`); } ``` -------------------------------- ### Error State Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-refresh-scraping.md Example of the error object stored on scrape failure. ```json { "date": "2025-05-20T10:30:00Z", "error": "Connection timeout", "scraper": "serpapi" } ``` -------------------------------- ### GET /api/keywords Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for fetching keywords for a domain, including integrated Search Console data. ```json { "keywords": [ { "ID": 1, "keyword": "example keyword", "device": "desktop", "country": "US", "domain": "example-com", "position": 5, "volume": 1200, "sticky": false, "history": {"2025-05-20": 5, "2025-05-21": 6}, "lastResult": [ {"position": 1, "url": "...", "title": "...", "skipped": false} ], "url": "https://example.com/page", "tags": ["tag1", "tag2"], "updating": false, "lastUpdateError": false, "lastUpdated": "2025-05-20T10:30:00Z", "added": "2025-03-01T00:00:00Z", "scData": { "impressions": {...}, "visits": {...}, "ctr": {...}, "position": {...} } } ] } ``` -------------------------------- ### Settings File Configuration Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md Example JSON structure for the encrypted settings file, showing various configuration options for scraping and notifications. ```json { scraper_type: 'serpapi', // Active scraper scaping_api: 'ENCRYPTED_KEY', // API key (encrypted) notification_interval: 'daily', // Notification frequency notification_email: 'owner@...', scrape_delay: '2000', // Delay between serial scrapes scrape_retry: true, // Retry failed scrapes scrape_strategy: 'smart', // Default strategy scrape_pagination_limit: 5, // Pages to scrape search_console: true, // SC integration enabled ... } ``` -------------------------------- ### GET /api/keyword Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for fetching a single keyword by its ID. ```json { "keyword": { "ID": 1, "keyword": "example", "position": 5, "history": {...}, ... } } ``` -------------------------------- ### Result Extraction - Example (HTML parser) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md An example implementation of the serpExtractor function using an HTML parser (cheerio). ```typescript serpExtractor: (content) => { const $ = cheerio.load(content); const results = []; $('h3').each((i, elem) => { const title = $(elem).text(); const url = $(elem).closest('a').attr('href'); if (title && url) { results.push({ title, url, position: i + 1 }); } }); return results; } ``` -------------------------------- ### Encryption with Cryptr Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Example of using Cryptr for encrypting and decrypting sensitive data in TypeScript. ```typescript const cryptr = new Cryptr(process.env.SECRET); // Encrypt on save const encrypted = cryptr.encrypt(plaintext); // Store encrypted // Decrypt on load const plaintext = cryptr.decrypt(encrypted); // Use plaintext ``` -------------------------------- ### Debugging Scrapers - Enable Logging Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md Example of error logs that appear in the console when scraping fails. ```text [ERROR] Scraping Keyword: example keyword [ERROR_MESSAGE]: {"status": 403, "message": "Invalid API key"} ``` -------------------------------- ### Custom Header Function Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md An example of a custom headers function for the SerpAPI scraper. ```typescript headers: (keyword, settings) => ({ 'X-API-KEY': decryptedApiKey, 'User-Agent': keyword.device === 'mobile' ? mobileAgent : desktopAgent }) ``` -------------------------------- ### Result Extraction - Example (JSON parser) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md An example implementation of the serpExtractor function using a JSON parser. ```typescript serpExtractor: (content) => { const data = JSON.parse(content); return data.results.map((result, i) => ({ title: result.title, url: result.url, position: i + 1 })); } ``` -------------------------------- ### Custom Scraper Integration - newscraper.ts example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md An example TypeScript file demonstrating how to define a custom scraper service. ```typescript import type { ScraperSettings, scraperExtractedItem } from '../../types'; const newscraper: ScraperSettings = { id: 'newscraper', name: 'New Scraper', website: 'https://newscraper.example.com', resultObjectKey: 'results', // Response JSON key with results allowsCity: false, nativePagination: false, headers: (keyword, settings) => { return { 'X-API-KEY': decryptedApiKey, 'User-Agent': keyword.device === 'mobile' ? mobileAgent : desktopAgent }; }, scrapeURL: (keyword, settings, countries, pagination) => { const start = pagination?.start || 0; const num = pagination?.num || 10; return `https://newscraper.example.com/search?q=${keyword.keyword}&start=${start}&num=${num}&api_key=${apiKey}`; }, serpExtractor: (content) => { const data = JSON.parse(content); return data.results.map((result, i) => ({ title: result.title, url: result.url, position: i + 1 })); } }; export default newscraper; ``` -------------------------------- ### Adding Custom Scraper Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md TypeScript example for defining a custom scraper service, including its settings and scraping logic. ```typescript const myScraper: ScraperSettings = { id: 'myscraper', name: 'My Scraper', website: 'https://myscraper.example.com', resultObjectKey: 'results', scrapeURL: (keyword, settings, countries, pagination) => { // Return API URL }, serpExtractor: (content) => { // Parse and return results } }; Register in `scrapers/index.ts`. ``` -------------------------------- ### SMTP Not Setup Properly Error Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/errors.md JSON response indicating that SMTP email notification settings are incomplete or incorrect. ```json { "success": false, "error": "SMTP has not been setup properly!" } ``` -------------------------------- ### Error Refreshing Keywords Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/errors.md Example JSON response for errors during the keyword refreshing process. ```json { "error": "keyword ID is Required!" | "When Refreshing all Keywords of a domian, the Domain name Must be provided." | "Error refreshing keywords!" } ``` -------------------------------- ### Last Result Storage Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/database-models.md Shows the JSON structure for storing the last SERP results, including skipped entries. ```json [ {"position": 1, "url": "https://...", "title": "...", "skipped": false}, {"position": 2, "url": "https://...", "title": "...", "skipped": false}, ... {"position": 100, "url": "", "title": "", "skipped": true} ] ``` -------------------------------- ### POST /api/dbmigrate Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response indicating successful database migration. ```json { "migrated": true } ``` -------------------------------- ### Custom Scraper Integration - Registering the scraper Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/scrapers.md Example of how to register a new custom scraper by importing and adding it to the allScrapers array. ```typescript import newscraper from './services/newscraper'; const allScrapers: ScraperSettings[] = [ // ... other scrapers newscraper ]; ``` -------------------------------- ### GET /api/refresh Error (400) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON error response for the GET /api/refresh endpoint. ```json { "error": "A Valid keyword, Country Code, and device is Required!" | "Scraper has not been set up yet." | "Error Scraping Search Results for the given keyword!" } ``` -------------------------------- ### POST /api/volume Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for updating search volume. ```json { "keywords": [...] } ``` -------------------------------- ### GET /api/keyword Error (400) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON error response for the GET /api/keyword endpoint. ```json { "error": "Keyword ID is Required!" | "Error Loading Keyword" } ``` -------------------------------- ### POST /api/keywords Response (201 Created) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for successfully adding new keywords. ```json { "keywords": [ { "ID": 1, "keyword": "example keyword", "position": 0, "updating": true, "history": {}, "tags": ["tag1", "tag2"], "lastUpdated": "2025-05-20T10:30:00Z", "added": "2025-05-20T10:30:00Z" } ] } ``` -------------------------------- ### Database Queries Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md Example TypeScript code for interacting with the database models (Domain and Keyword) to find, update, and delete records. ```typescript // Get domain with keywords const domain = await Domain.findOne({ where: { domain: 'example.com' } }); const keywords = await Keyword.findAll({ where: { domain: domain.slug } }); // Update keyword position await keyword.update({ position: 5, history: JSON.stringify(updatedHistory), lastUpdated: new Date().toJSON(), updating: false }); // Delete domain and keywords await Keyword.destroy({ where: { domain: 'example-com' } }); await Domain.destroy({ where: { domain: 'example.com' } }); ``` -------------------------------- ### PUT /api/keywords Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for a successful keyword update. ```json { "keywords": [...] } ``` -------------------------------- ### Keyword State Representation Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md Example of how keyword state is represented, including position, updating status, and error information. ```json { "2025-05-20": 5, "2025-05-21": 6, "2025-05-22": 5 } ``` -------------------------------- ### POST /api/ideas Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for generating new keyword ideas. ```json { "keywords": [ { "uid": "...", "keyword": "generated keyword", "competition": "LOW", "competitionIndex": 25, "avgMonthlySearches": 500, "country": "US", "domain": "example-com", "added": 1716238200000, "updated": 1716238200000, "position": 0 } ] } ``` -------------------------------- ### Last Update Error Storage Example Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/database-models.md Demonstrates the JSON format for storing error information during updates, or the string 'false'. ```json { "date": "2025-05-20T10:30:00Z", "error": "Connection timeout", "scraper": "serpapi" } ``` -------------------------------- ### Docker Deployment Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Dockerfile for setting up a Docker container for the application. ```dockerfile FROM node:18 WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build EXPOSE 3000 CMD ["npm", "start"] ``` -------------------------------- ### Required Environment Variables Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md List of essential environment variables required for deployment. ```bash USER=admin PASSWORD=secure_password SECRET=min32charssecretkey SEARCH_CONSOLE_CLIENT_EMAIL=optional SEARCH_CONSOLE_PRIVATE_KEY=optional ``` -------------------------------- ### GET /api/adwords Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for Google Ads OAuth redirect URL. ```json { "redirectUrl": "https://accounts.google.com/o/oauth2/v2/auth?..." } ``` -------------------------------- ### Docker Deployment Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md Docker commands for building and running Serpbear. ```bash docker build -t serpbear . docker run -p 3000:3000 -e SECRET=... -e PASSWORD=... serpbear ``` -------------------------------- ### Run Migrations Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/database-models.md Command to run database migrations. ```bash npm run db:migrate ``` -------------------------------- ### Testing Commands Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md npm commands for running tests in different modes. ```bash npm run test # Watch mode npm run test:ci # CI mode npm run test:cv # With coverage ``` -------------------------------- ### Search Console Integration Flow Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Diagram detailing the flow for integrating with Google Search Console, including credential handling, API calls, and data caching. ```text User adds domain with SC credentials ↓ PUT /api/domains { search_console: {...} } ↓ checkSearchConsoleIntegration() validates credentials ↓ Encrypt credentials with SECRET ↓ Store in Domain.search_console (JSON) ↓ GET /api/keywords?domain=example.com ↓ readLocalSCData() check cache ├─ Cache valid (< 24h) → Return cached data └─ Cache invalid → fetchDomainSCData() ↓ fetchDomainSCData() queries Google API ├─ Build JWT with service account ├─ Query searchanalytics.list API ├─ Aggregate by time period ├─ Cache to data/searchconsole/{domain}.json └─ Return data ↓ integrateKeywordSCData() merges SC data into keywords ↓ Return enriched keywords with scData field ``` -------------------------------- ### Core Modules Dependencies Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Dependency diagram for core backend modules, including API routes, database connections, and models. ```text pages/api/*.ts (Next.js routes) ↓ ├─ verifyUser() → JWT auth ├─ database.sync() → DB connection ├─ getAppSettings() → Settings loader └─ Domain/Keyword models → Sequelize queries Domain/Keyword models ↓ database/database.ts (Sequelize instance) ↓ SQLite via sqlite3 driver Keyword operations ↓ ├─ refresh.ts (position updates) ├─ scraper.ts (SERP scraping) └─ searchConsole.ts (SC integration) Scraper selection ↓ scrapers/index.ts (registry) ↓ scrapers/services/*.ts (individual scrapers) ``` -------------------------------- ### Database Backup Command Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md Command to create a backup of the SQLite database file. ```bash cp data/database.sqlite data/database.backup.sqlite ``` -------------------------------- ### Strategy-Based Scraping Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Detailed diagram of the `scrapeKeywordWithStrategy` function, outlining different scraping approaches based on strategy, position, and settings. ```text scrapeKeywordWithStrategy() ├─ Resolve domain strategy override ├─ Resolve pagination limit ├─ Resolve smart fallback setting │ └─ If strategy = "basic" └─ Scrape page 1 only (10 results) ├─ If strategy = "custom" └─ Scrape pages 1-N (per paginationLimit) └─ If strategy = "smart" ├─ If position = 0 (new) │ └─ Scrape page 1 └─ If position > 0 (known) ├─ Calculate target page ├─ Scrape target page ± 1 └─ If smartFullFallback └─ Walk remaining pages if not found ↓ Build full 100-position result array ├─ Scraped positions: {position, url, title} └─ Unscraped positions: {position, url: "", title: "", skipped: true} ↓ Find domain URL in results (with subdomain matching) ↓ Return RefreshResult {position, url, result[], error?} ``` -------------------------------- ### getKeywordsVolume Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-helpers.md Fetch search volume for keywords from Google Ads. ```typescript export const getKeywordsVolume = async ( keywords: KeywordType[] ): Promise<{volumes: {[id: number]: number}|false}> ``` -------------------------------- ### Feature Modules Dependencies Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Dependency diagram for feature-specific modules, such as Search Console, Google Ads, Email Notifications, and Scheduled Tasks. ```text Search Console ├─ utils/searchConsole.ts ├─ services/searchConsole.ts └─ pages/api/searchconsole.ts ├─ readLocalSCData() ├─ fetchDomainSCData() └─ integrateKeywordSCData() Feature Modules ├─ utils/adwords.ts ├─ pages/api/adwords.ts └─ pages/api/ideas.ts Email Notifications ├─ utils/generateEmail.ts ├─ pages/api/notify.ts └─ Nodemailer (SMTP) Scheduled Tasks ├─ cron.js (entry point) └─ Croner (job scheduling) ``` -------------------------------- ### Settings Persistence Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Flow for saving and retrieving user settings, including encryption of sensitive fields. ```text User updates settings ↓ PUT /api/settings { settings: {...} } ↓ Encrypt sensitive fields: ├─ scaping_api ├─ smtp_password ├─ search_console_* └─ adwords_* ↓ Write to data/settings.json ↓ GET /api/settings ↓ safeReadJSON() loads with error handling ↓ Decrypt sensitive fields ↓ Return to client ``` -------------------------------- ### Directory Structure Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md The file and directory structure of the SerpBear project, outlining the organization of API routes, database models, utility functions, scrapers, components, and configuration files. ```bash /workspace/home/serpbear/ ├── pages/ │ ├── api/ # API routes (Next.js API handlers) │ │ ├── domains.ts # Domain CRUD │ │ ├── keywords.ts # Keyword CRUD │ │ ├── refresh.ts # Scrape & refresh │ │ ├── settings.ts # Global settings │ │ ├── searchconsole.ts # SC integration │ │ ├── insight.ts # Analytics │ │ ├── login.ts # Authentication │ │ ├── adwords.ts # Google Ads │ │ ├── notify.ts # Email notifications │ │ └── ... │ ├── index.tsx # Home page │ ├── _app.tsx # App wrapper │ └── _document.tsx # HTML document ├── database/ │ ├── models/ │ │ ├── domain.ts # Domain model │ │ └── keyword.ts # Keyword model │ ├── database.ts # Sequelize instance │ ├── migrations/ # DB migrations │ └── config.js # Sequelize config ├── utils/ │ ├── refresh.ts # Keyword refresh logic │ ├── scraper.ts # Scraping core │ ├── searchConsole.ts # SC utilities │ ├── adwords.ts # Google Ads integration │ ├── parseKeywords.ts # Data parsing │ ├── verifyUser.ts # JWT verification │ ├── domains.ts # Domain stats │ ├── countries.ts # Geographic data │ ├── generateEmail.ts # Email templating │ ├── client/ # Browser-side utilities │ │ ├── exportcsv.ts │ │ ├── helpers.ts │ │ └── sortFilter.ts │ └── ... ├── services/ │ ├── searchConsole.ts # SC service layer │ └── settings.ts # Settings service ├── scrapers/ │ ├── index.ts # Scraper registry │ └── services/ │ ├── serpapi.ts │ ├── searchapi.ts │ ├── serper.ts │ ├── proxy.ts │ └── ... # 10+ scraper implementations ├── components/ # React components ├── hooks/ # Custom React hooks ├── types.d.ts # Global type definitions ├── data/ # Runtime data │ ├── database.sqlite # SQLite database │ ├── settings.json # Encrypted settings │ ├── failed_queue.json # Retry queue │ └── searchconsole/ # SC cache ├── cron.js # Scheduled tasks ├── package.json ├── next.config.js ├── tsconfig.json └── tailwind.config.js ``` -------------------------------- ### Error Loading Settings Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/errors.md JSON response when there is an issue loading the application settings. ```json { "error": "Error Loading Settings!" } ``` -------------------------------- ### GET /api/insight Error (400) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Error response for aggregated Search Console insights. ```json { "data": null, "error": "Google Search Console is not Integrated." | "Error Fetching Stats from Google Search Console." } ``` -------------------------------- ### GET /api/domain Error (400) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Error response when fetching a single domain fails. ```json { "error": "Domain Name is Required!" | "Error Loading Domain" } ``` -------------------------------- ### Cron Job Pseudocode Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Pseudocode illustrating the background cron job for refreshing keywords, SC data, and retrying failed tasks. ```javascript // Pseudocode croner(schedule, () => { // Refresh all keywords const domains = await Domain.findAll(); for (const domain of domains) { const keywords = await Keyword.findAll({ where: { domain: domain.slug } }); await refreshAndUpdateKeywords(keywords, settings); } // Refresh Search Console data for (const domain of domains) { await fetchDomainSCData(domain, scAPI); } // Send email notifications await sendNotifications(); // Retry failed keywords const failedQueue = await readFailedQueue(); for (const keywordId of failedQueue) { await refreshAndUpdateKeyword(keywordId); } }); ``` -------------------------------- ### POST /api/volume Request Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON request body for updating search volume for keywords. ```json { "keywords": [1, 2, 3], "domain": "example-com", "update": true } ``` -------------------------------- ### POST /api/ideas Request Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON request body for generating new keyword ideas. ```json { "domain": "example-com", "settings": { "seedSCKeywords": true, "seedCurrentKeywords": true, "seedDomain": true, "language": "en", "countries": ["US"], "keywords": "seed keyword" } } ``` -------------------------------- ### DELETE /api/keywords Error (400) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON error response for the DELETE /api/keywords endpoint. ```json { "error": "keyword ID is Required!" | "Could Not Remove Keyword!" } ``` -------------------------------- ### Adding Keywords Flow Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Data flow diagram illustrating the process of adding new keywords, from frontend form submission to background updates. ```text Frontend Form (add keywords) ↓ POST /api/keywords ↓ API Handler ├─ Parse request body ├─ Create Keyword rows in DB ├─ Set updating: true, position: 0 ├─ Parse tags from CSV ├─ Queue for immediate refresh (async) ├─ Fetch volume from Google Ads (if configured) └─ Return new keywords ↓ Frontend displays new keywords with spinner ↓ Cron/Background: refreshAndUpdateKeywords() ├─ Load scraper settings ├─ Scrape Google for each keyword ├─ Update DB: position, history, lastResult └─ Mark updating: false ↓ Frontend re-fetches keywords table (react-query) ↓ Updated positions display ``` -------------------------------- ### DELETE /api/keywords Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for successfully deleting keywords. ```json { "keywordsRemoved": 3 } ``` -------------------------------- ### Authentication Flow Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md A step-by-step flow diagram for user authentication within the application. ```text User Request ↓ Next.js API Handler ↓ verifyUser() checks JWT cookie ↓ ├─ Valid → Proceed └─ Invalid → 401 Unauthorized ``` -------------------------------- ### PUT /api/keywords Error (400) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON error response for the PUT /api/keywords endpoint. ```json { "error": "keyword ID is Required!" | "keyword Payload Missing!" | "Invalid Payload!" } ``` -------------------------------- ### Settings Data Not Provided Error Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/errors.md JSON response when the request body for updating settings is missing the required data. ```json { "error": "Settings Data not Provided!" } ``` -------------------------------- ### POST /api/keywords Error (400) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON error response for the POST /api/keywords endpoint. ```json { "error": "Necessary Keyword Data Missing" | "Could Not Add New Keyword!" } ``` -------------------------------- ### POST /api/keywords Request Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON request body for adding new keywords to a domain. ```json { "keywords": [ { "keyword": "example keyword", "device": "desktop", "country": "US", "domain": "example-com", "tags": "tag1, tag2", "city": "" } ] } ``` -------------------------------- ### Keyword Refresh Flow Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Diagram illustrating the process of refreshing keywords, including API calls, database operations, and scraping strategies. ```text User clicks "Refresh" ↓ POST /api/refresh { id: [1,2,3], domain: "example.com" } ↓ verifyUser() authorization check ↓ Load keywords from DB ↓ Load scraper settings ↓ Resolve domain-level strategy overrides ↓ ├─ Native pagination → Parallel scrape all keywords └─ Manual pagination → Serial scrape with delay ↓ For each keyword: ├─ scrapeKeywordWithStrategy() ├─ Extract results (100-position array) ├─ Find domain URL in results ├─ Update position in DB ├─ Update history └─ Update lastResult ↓ Return updated keywords ↓ Frontend updates table ``` -------------------------------- ### Required Environment Variables for Production Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md These environment variables are essential for running SerpBear in a production environment. ```bash USER=admin PASSWORD=secure_password123 SECRET=very_long_secret_key_min_32_chars SEARCH_CONSOLE_CLIENT_EMAIL=optional SEARCH_CONSOLE_PRIVATE_KEY=optional ``` -------------------------------- ### GET /api/insight Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Successful response for aggregated Search Console insights. ```json { "data": { "stats": [ {"date": "2025-05-20", "clicks": 10, "impressions": 100, "ctr": 0.1, "position": 15} ], "keywords": [ {"keyword": "...", "clicks": 5, "impressions": 50, "ctr": 0.1, "position": 10} ], "countries": [ {"country": "US", "clicks": 8, "impressions": 80} ], "pages": [ {"page": "https://example.com", "clicks": 5, "impressions": 50} ] } } ``` -------------------------------- ### Search Console Caching Logic Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md TypeScript snippet showing the logic for caching Search Console data for 24 hours. ```typescript if (localSCData && fetchTimeDiff <= 86400000) { return localSCData; // Return cached } // Otherwise fetch fresh ``` -------------------------------- ### GET /api/searchconsole Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Successful response for fetching Search Console data. ```json { "data": { "threeDays": [...], "sevenDays": [...], "thirtyDays": [...], "lastFetched": "2025-05-20T10:30:00Z", "stats": [...] } } ``` -------------------------------- ### GET /api/domains Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Successful response fetching all domains with optional statistics. ```json { "domains": [ { "ID": 1, "domain": "example.com", "slug": "example-com", "keywordCount": 42, "lastUpdated": "2025-05-20T10:30:00Z", "added": "2025-03-01T00:00:00Z", "tags": "[]", "notification": true, "notification_interval": "daily", "notification_emails": "owner@example.com", "avgPosition": 15.2, "search_console": "{}" } ] } ``` -------------------------------- ### Data Model Entities Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Diagram illustrating the core entities of the data model: Domain and Keyword, including their attributes and relationships. ```text ┌─────────────────────────┐ │ Domain │ ├─────────────────────────┤ │ ID (PK) │ │ domain (UNIQUE) │ │ slug (UNIQUE) │ │ notification settings │ │ Search Console config │ │ scrape strategy │ │ tags │ │ statistics │ └────────────┬────────────┘ │ 1 │ │ many ↓ ┌─────────────────────────┐ │ Keyword │ ├─────────────────────────┤ │ ID (PK) │ │ keyword (text) │ │ domain (FK → Domain) │ │ device (desktop/mobile) │ │ country (ISO code) │ │ position (SERP rank) │ │ history (JSON) │ │ lastResult (JSON) │ │ volume (search vol) │ │ tags (JSON) │ │ sticky (pinned) │ │ updating (state) │ │ lastUpdateError (JSON) │ └─────────────────────────┘ ``` -------------------------------- ### POST /api/cron Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for triggering a scheduled refresh of keywords. ```json { "started": true } ``` -------------------------------- ### Required Environment Variables Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/README.md Environment variables required for running Serpbear, including login credentials and JWT secret. ```bash USER=admin # Login username PASSWORD=secure_password # Login password SECRET=min32charssecretkey # JWT signing key ``` -------------------------------- ### getAdwordsKeywordIdeas Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-helpers.md Generate keyword ideas from Google Ads. ```typescript export const getAdwordsKeywordIdeas = async ( domain: DomainType, settings: DomainIdeasSettings ): Promise ``` -------------------------------- ### PUT /api/keywords Request Body (Tags) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON request body for updating the tags of keywords. ```json { "tags": { "1": ["newtag1", "newtag2"], "2": ["newtag3"] } } ``` -------------------------------- ### Structured API Error Response Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Example of a structured JSON error response format for API endpoints. ```json { "error": "message" } ``` -------------------------------- ### GET /api/settings Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Successful response for application settings with decrypted sensitive values. ```json { "settings": { "scraper_type": "serpapi", "scaping_api": "YOUR_API_KEY", "notification_interval": "daily", "notification_email": "owner@example.com", "notification_email_from": "noreply@example.com", "smtp_server": "smtp.example.com", "smtp_port": "587", "smtp_username": "user@example.com", "smtp_password": "PASSWORD", "scrape_delay": "2000", "scrape_retry": true, "scrape_strategy": "basic", "scrape_pagination_limit": 5, "scrape_smart_full_fallback": false, "search_console": true, "search_console_client_email": "...", "search_console_private_key": "...", "adwords_client_id": "...", "adwords_developer_token": "...", "available_scrapers": [ {"label": "SerpAPI", "value": "serpapi", "allowsCity": false} ], "failed_queue": [], "version": "3.1.0" } } ``` -------------------------------- ### parseKeywords() Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-helpers.md Parse raw keyword database objects into display-ready format. Converts keyword database records to application types, parsing JSON fields, generating unique UIDs, and filtering null entries. ```typescript const parseKeywords = (keywords: any[]): KeywordType[] ``` ```typescript import parseKeywords from './utils/parseKeywords'; const rawKeywords = await Keyword.findAll({ where: { domain: 'example-com' } }); const keywords = parseKeywords(rawKeywords.map(k => k.get({ plain: true }))); console.log(keywords[0].history); // {date: position} object, not JSON string ``` -------------------------------- ### PUT /api/clearfailed Response (200 OK) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON response for clearing the failed keywords retry queue. ```json { "cleared": true } ``` -------------------------------- ### PUT /api/keywords Request Body (Sticky) Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/endpoints.md Example JSON request body for updating the sticky flag of a keyword. ```json { "sticky": true } ``` -------------------------------- ### updateKeywordsVolumeData Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/utils-helpers.md Save search volumes to database. ```typescript export const updateKeywordsVolumeData = async ( volumes: {[id: number]: number} ): Promise ``` -------------------------------- ### JWT Token Flow Source: https://github.com/towfiqi/serpbear/blob/main/_autodocs/architecture.md Illustrates the process of obtaining and using JWT tokens for authentication, including API requests and verification. ```text POST /api/login { username, password } Verify against USER/USER_NAME env var Verify against PASSWORD env var jwt.sign({ user: username }, SECRET) Set HTTP-only cookie "token" ├─ sameSite: 'lax' ├─ httpOnly: true └─ maxAge: SESSION_DURATION hours Return { success: true } Protected API request verifyUser() checks cookie ├─ Extract "token" from cookies ├─ jwt.verify(token, SECRET) └─ Return "authorized" or error Proceed with request or 401 ```