### Install Go using Homebrew Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Install the Go programming language using Homebrew. Verify the installation with `go version`. ```bash brew install go ``` -------------------------------- ### Start Local SaaS Development Environment Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/saas.md Starts a local development environment for the SaaS edition, including a Postgres database and hot-reloading. Access the admin UI at http://localhost:8080/admin with default credentials. ```bash make saas-dev # Start local dev environment (Postgres + hot reload) # Visit http://localhost:8080/admin (admin / 1234#abcd) ``` -------------------------------- ### Start Docker Web UI Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md This command starts the web interface by creating a data directory and running the Docker container, exposing the web UI on port 8080. ```bash mkdir -p gmapsdata docker run \ -v "$PWD/gmapsdata:/gmapsdata" \ -p 8080:8080 \ gosom/google-maps-scraper \ -data-folder /gmapsdata ``` -------------------------------- ### Install tsx for TypeScript Scraper Source: https://github.com/gosom/google-maps-scraper/blob/main/examples/examples-api/README.md Install the tsx package globally to run TypeScript examples. This is a one-time setup step. ```bash npm install -g tsx ``` -------------------------------- ### Make Start Script Executable Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Grant execute permissions to the `start.sh` script within the application bundle. ```bash chmod +x GoogleMapsScraper.app/Contents/MacOS/start.sh ``` -------------------------------- ### Install Multipass Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/development-saas.md Installs the Multipass hypervisor on your system. This is a prerequisite for creating local VMs. ```bash sudo snap install multipass ``` -------------------------------- ### Install Homebrew on macOS Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Run this command in the terminal to install Homebrew. Ensure Homebrew is added to your PATH after installation. ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` ```bash echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zshrc source ~/.zshrc ``` -------------------------------- ### Start Google Maps Scraper REST API Server Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/recipes.md Start the Web UI/API server for programmatic access to the scraper. Data will be stored in the specified folder. ```bash mkdir -p gmapsdata docker run \ -v gmaps-playwright-cache:/opt \ -v "$PWD/gmapsdata:/gmapsdata" \ -p 8080:8080 \ gosom/google-maps-scraper \ -data-folder /gmapsdata ``` -------------------------------- ### Build and Run from Source Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Clone the repository, download Go dependencies, build the executable, and run the scraper from source. Ensure Go 1.26.4+ is installed. The first run downloads Playwright browser libraries. ```bash git clone https://github.com/gosom/google-maps-scraper.git cd google-maps-scraper go mod download go build ./google-maps-scraper -input example-queries.txt -results results.csv -exit-on-inactivity 3m ``` -------------------------------- ### Set Environment Variables Source: https://github.com/gosom/google-maps-scraper/blob/main/examples/examples-api/README.md Set the BASE_URL and API_KEY environment variables before running any examples. ```bash export BASE_URL="https://your-instance.example.com" export API_KEY="gms_your_api_key_here" ``` -------------------------------- ### Create DMG Installer for macOS App Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Prepare the application for distribution by creating a disk image (.dmg) file. This involves copying the app to a temporary directory and using `hdiutil` to create the DMG. ```bash mkdir -p ~/dmg-tmp/GoogleMapsScraper cp -R /path/to/GoogleMapsScraper.app ~/dmg-tmp/GoogleMapsScraper/ ``` ```bash hdiutil create -volname "GoogleMapsScraper" -srcfolder ~/dmg-tmp/GoogleMapsScraper -ov -format UDZO GoogleMapsScraper.dmg ``` -------------------------------- ### Combined Google Places Search Example Source: https://github.com/gosom/google-maps-scraper/blob/main/netnut.md This example demonstrates how to combine multiple parameters for a precise Google Places search. It searches for 'coffee' with specific language, region, and location targeting. ```url https://serp-api.netnut.io/search?engine=google_places&q=coffee&hl=en&gl=us&uule=w+CAIQICINVW5pdGVkIFN0YXRlcw&udm=1 ``` -------------------------------- ### Start PostgreSQL for Distributed Scraping Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Initiate a PostgreSQL database using Docker Compose for distributed scraping jobs. ```bash docker-compose -f docker-compose.dev.yaml up -d ``` -------------------------------- ### Install AI Agent Skill Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Install the Google Maps Scraper skill for use with AI coding agents. This command adds the skill to your agent's toolkit. ```bash npx skills add gosom/google-maps-scraper ``` -------------------------------- ### Create Application Start Script Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Create a shell script to run the scraper in the background, open a browser, and manage the scraper's lifecycle. This script is placed inside the application's MacOS directory. ```bash #!/bin/bash DIR="$(cd "$(dirname "$0")" && pwd)" cd "$HOME" # Run scraper in background and get its PID "$DIR/google_maps_scraper" > "$HOME/google_maps_scraper.log" 2>&1 & SCRAPER_PID=$! # Open browser sleep 2 open http://localhost:8080 # Idle timeout in seconds IDLE_TIMEOUT=6000 # Wait and kill sleep $IDLE_TIMEOUT # Check if still running, and kill it if ps -p $SCRAPER_PID > /dev/null; then echo "Stopping scraper after $IDLE_TIMEOUT seconds of idle time." kill $SCRAPER_PID fi ``` -------------------------------- ### Configure Proxies for Scraping Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Use proxies to avoid rate limiting during large scraping jobs. This example shows how to specify a list of proxies with different protocols. ```bash ./google-maps-scraper \ -input queries.txt \ -results results.csv \ -proxies 'socks5://user:pass@host:port,http://host2:port2' \ -depth 1 -c 2 ``` -------------------------------- ### Configure 8 Concurrent Jobs with 2 Browsers and 4 Tabs Each Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md This example demonstrates how to run 8 concurrent scraping jobs, utilizing 2 browser processes, with each browser managing 4 page tabs. This configuration aims to increase throughput by sharing browser resources. ```bash ./google-maps-scraper \ -c 8 \ -browser-pool-size 2 \ -pages-per-browser 4 \ -input queries.txt \ -results results.csv \ -depth 1 ``` -------------------------------- ### Connect to Worker via SSH Source: https://github.com/gosom/google-maps-scraper/blob/main/admin/templates/workers.html Example command to connect to a provisioned worker instance via SSH using the downloaded private key. Assumes the key is named 'gmapspro_id_ed25519' and the worker IP is ''. ```bash ssh -p 2222 -i gmapspro_id_ed25519 root@ ``` -------------------------------- ### Custom Input IDs Example Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Define custom identifiers for your input queries by appending '#!#' followed by your desired ID to the business name. ```text Matsuhisa Athens #!#MyCustomID ``` -------------------------------- ### Get VM Information Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/development-saas.md Retrieves detailed information about a Multipass VM, including its IP address. Use the VM's name as an argument. ```bash multipass info mapsdev ``` -------------------------------- ### Search Coffee Shops in the US using JavaScript Source: https://github.com/gosom/google-maps-scraper/blob/main/netnut.md A JavaScript example demonstrating how to fetch Google Maps Places API results for 'coffee' in the US. It utilizes the Fetch API and constructs the authorization header using Base64 encoding. ```javascript const params = new URLSearchParams({ engine: "google_places", q: "coffee", gl: "us", hl: "en" }); const response = await fetch(`https://serp-api.netnut.io/search?${params}`, { method: "GET", headers: { "Authorization": "Basic " + Buffer.from("username:password").toString("base64") } }); const data = await response.json(); console.log(data); ``` -------------------------------- ### Example Google Maps Search Queries Source: https://github.com/gosom/google-maps-scraper/blob/main/skills/google-maps-scraper/SKILL.md Generate specific search queries for Google Maps, one per line, targeting different neighborhoods within a city for better coverage. This helps in breaking down broad searches into manageable and effective queries. ```text dentists in Berlin Mitte dentists in Berlin Kreuzberg dentists in Berlin Charlottenburg dentists in Berlin Prenzlauer Berg dentists in Berlin Friedrichshain dentists in Berlin Neukölln dentists in Berlin Schöneberg dentists in Berlin Tempelhof ``` -------------------------------- ### Initialize Terminal and WebSocket Connection Source: https://github.com/gosom/google-maps-scraper/blob/main/admin/templates/terminal.html Sets up the terminal, loads the FitAddon for resizing, and establishes a WebSocket connection to the worker. Handles initial connection, data transfer, and size synchronization. ```javascript (function() { var workerID = {{.WorkerID}}; var workerName = "{{.WorkerName}}"; var term = new Terminal({ cursorBlink: true, fontSize: 14, fontFamily: "'Fira Code', 'Cascadia Code', 'Source Code Pro', 'Menlo', monospace", theme: { background: '#0d1117', foreground: '#c9d1d9', cursor: '#58a6ff', selectionBackground: '#264f78' } }); var fitAddon = new FitAddon.FitAddon(); term.loadAddon(fitAddon); var termEl = document.getElementById('terminal'); var statusEl = document.getElementById('status'); var statusText = document.getElementById('statusText'); term.open(termEl); fitAddon.fit(); var protocol = location.protocol === 'https:' ? 'wss:' : 'ws:'; var wsURL = protocol + '//' + location.host + '/admin/workers/' + workerID + '/terminal/ws'; var ws = new WebSocket(wsURL); ws.binaryType = 'arraybuffer'; function sendSize() { var size = JSON.stringify({high: term.rows, width: term.cols}); var buf = new TextEncoder().encode(size); if (ws.readyState === WebSocket.OPEN) { ws.send(buf.buffer); } } ws.onopen = function() { statusText.textContent = 'Connected to ' + workerName; statusEl.className = 'connected'; sendSize(); }; ws.onmessage = function(ev) { if (typeof ev.data === 'string') { term.write(ev.data); } else { term.write(new Uint8Array(ev.data)); } }; ws.onclose = function() { statusText.textContent = 'Disconnected from ' + workerName; statusEl.className = 'error'; term.write('\r\n\r\n--- Connection closed ---\r\n'); }; ws.onerror = function() { statusText.textContent = 'Connection error'; statusEl.className = 'error'; }; term.onData(function(data) { if (ws.readyState === WebSocket.OPEN) { ws.send(data); } }); window.addEventListener('resize', function() { fitAddon.fit(); sendSize(); }); })(); ``` -------------------------------- ### Launch VM with Cloud-init Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/development-saas.md Launches a new Multipass VM named 'dev' and configures it using the specified cloud-init file. Ensure the cloud-init file exists in the current directory. ```bash multipass launch -n dev --cloud-init cloud-init.yaml ``` -------------------------------- ### Run golangci-lint with Project Configuration Source: https://github.com/gosom/google-maps-scraper/blob/main/AGENTS.md Perform static analysis on the project using golangci-lint with the project's specific configuration. This helps in enforcing code quality and identifying potential issues. ```bash make lint ``` -------------------------------- ### Prepare Queries for Google Maps Scraper Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/recipes.md Create a text file with natural Google Maps search queries. Each line should contain a business type, city, region, and country for effective lead generation. ```bash cat > example-queries.txt <<'EOF' dentists in Berlin Germany plumbers in Austin Texas coffee shops in Dublin Ireland EOF ``` -------------------------------- ### Run macOS Application Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Open the created macOS application bundle using the `open` command. ```bash open GoogleMapsScraper.app ``` -------------------------------- ### Get Job Results Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/saas.md Retrieve the results of a specific scraping job using its job ID. ```APIDOC ## GET /api/v1/jobs/{job_id} ### Description Retrieve the results of a specific scraping job using its job ID. ### Method GET ### Endpoint /api/v1/jobs/{job_id} ### Parameters #### Path Parameters - **job_id** (string) - Required - The ID of the job to retrieve. ### Request Example ```bash curl https://your-server/api/v1/jobs/some-unique-job-id \ -H "X-API-Key: your-api-key" ``` ### Response #### Success Response (200) - **results** (object) - The scraping results for the job. #### Response Example ```json { "results": { "businesses": [ { "name": "Example Business", "address": "123 Main St" } ] } } ``` ``` -------------------------------- ### Create VM for Testing Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/development-saas.md Launches a Multipass VM named 'mapsdev' using a local cloud-init configuration file. This is a specific step for testing provisioning. ```bash multipass launch -n mapsdev --cloud-init local/cloud-init-test.yaml ``` -------------------------------- ### Get job details Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Fetches detailed information about a specific scraping job using its unique identifier. ```APIDOC ## GET /api/v1/jobs/{id} ### Description Get job details. ### Method GET ### Endpoint /api/v1/jobs/{id} ### Parameters #### Path Parameters - **id** (string) - Required - The unique identifier of the job. ### Request Example (No request body mentioned in the source) ### Response #### Success Response (200) (No specific fields mentioned in the source, refer to OpenAPI documentation for details) #### Response Example (Refer to OpenAPI documentation for example) ``` -------------------------------- ### Configure Proxies via Command Line Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/proxies.md Use the -proxies flag with a comma-separated list of proxy URLs. Supported protocols include socks5, socks5h, http, and https. ```bash ./google-maps-scraper \ -input queries.txt \ -results results.csv \ -proxies "socks5://user:pass@host:port,http://host2:port2" \ -depth 1 ``` -------------------------------- ### Build Go Project for macOS Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Compile the Go project for macOS using the `go build` command. This creates an executable file. ```bash go build -o google_maps_scraper ``` -------------------------------- ### Create Feature Branch for Merge Source: https://github.com/gosom/google-maps-scraper/blob/main/migration-pro.md Before starting the merge process, create a new Git branch to isolate the changes. ```bash git checkout -b feature/merge-saas ``` -------------------------------- ### Build and Test Commands Source: https://github.com/gosom/google-maps-scraper/blob/main/migration-pro.md Commands to verify the build and run tests for the project. Use these after updating dependencies and making code changes. ```bash go build ./... ``` ```bash go build -o bin/google-maps-scraper . ``` ```bash go build -o bin/gmapssaas ./cmd/gmapssaas/ ``` ```bash go test ./... ``` -------------------------------- ### Pull Docker Image Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Pull the official Docker image for Google Maps Scraper. This is the recommended installation method for using the scraper with Docker. ```bash docker pull gosom/google-maps-scraper ``` -------------------------------- ### Create Queries File for Grid Search Source: https://github.com/gosom/google-maps-scraper/blob/main/skills/google-maps-scraper/SKILL.md Prepare a file containing only the search term for grid searches. The grid functionality handles location qualifiers, so they should be omitted from the queries file. ```bash echo "dentists" > /tmp/gmaps_queries.txt ``` -------------------------------- ### Get Specific Job Results via REST API Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/saas.md Fetches the results for a specific scraping job using its ID. Requires an API key in the 'X-API-Key' header. ```bash # Get job results curl https://your-server/api/v1/jobs/{job_id} \ -H "X-API-Key: your-api-key" ``` -------------------------------- ### Execute Comprehensive Grid Search with Docker Source: https://github.com/gosom/google-maps-scraper/blob/main/skills/google-maps-scraper/SKILL.md This command demonstrates a comprehensive grid search for dentists across Berlin using Docker. It maps necessary volumes, specifies the query and results files, and configures grid search parameters like bounding box and cell size. ```bash docker rm gmaps-scraper 2>/dev/null docker run \ --name gmaps-scraper \ -v gmaps-playwright-cache:/opt \ -v /tmp/gmaps_queries.txt:/queries.txt \ -v /tmp/gmaps_dentists_berlin.csv:/results.csv \ gosom/google-maps-scraper \ -input /queries.txt \ -results /results.csv \ -exit-on-inactivity 3m \ -depth 5 \ -grid-bbox "52.34,13.09,52.68,13.76" \ -grid-cell 1.0 ``` -------------------------------- ### Cloud-init Configuration Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/development-saas.md Defines the cloud-init configuration for the VM, specifically setting up authorized SSH keys. Replace 'your_public_key' with your actual public SSH key. ```yaml ssh_authorized_keys: - "ssh-ed25519 AAAA... your_public_key ..." ``` -------------------------------- ### Go Scraper: Keywords as Arguments Source: https://github.com/gosom/google-maps-scraper/blob/main/examples/examples-api/README.md Run the Go scraper with keywords provided directly as command-line arguments. Requires Go 1.21+. ```bash go run go/main.go -base-url "$BASE_URL" -api-key "$API_KEY" "cafes in athens" "hotels in berlin" ``` -------------------------------- ### Configure Proxies with Docker Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/proxies.md When using Docker, mount your query file and output directory, and specify proxies using the -proxies flag within the docker run command. Ensure to map the cache volume for Playwright. ```bash mkdir -p gmaps-output docker run \ -v gmaps-playwright-cache:/opt \ -v "$PWD/example-queries.txt:/queries.txt:ro" \ -v "$PWD/gmaps-output:/out" \ gosom/google-maps-scraper \ -input /queries.txt \ -results /out/results.csv \ -depth 1 \ -proxies "http://user:pass@host:port,socks5://host:port" \ -exit-on-inactivity 3m ``` -------------------------------- ### Clone and Navigate to Repository Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Clone the google-maps-scraper repository and change the directory to the cloned repository. ```bash git clone https://github.com/gosom/google-maps-scraper.git cd google-maps-scraper ``` -------------------------------- ### Create Info.plist for macOS App Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Add an `Info.plist` file to the application bundle to define metadata for the macOS application, specifying the executable script. ```xml CFBundleExecutable start.sh CFBundleIdentifier com.yourcompany.googlemapsscraper CFBundleName Google Maps Scraper CFBundleVersion 1.0 ``` -------------------------------- ### Create macOS Application Structure Source: https://github.com/gosom/google-maps-scraper/blob/main/MacOS instructions.md Manually create the necessary directory structure for a macOS application bundle and move the compiled executable into it. ```bash mkdir -p GoogleMapsScraper.app/Contents/MacOS mv google_maps_scraper GoogleMapsScraper.app/Contents/MacOS/ ``` -------------------------------- ### Google Maps Scraper Command Line Options Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Lists all available command-line options for configuring the google-maps-scraper. Includes core options, email/review settings, location parameters, web server, database, proxy, export, and advanced features. ```bash Usage: google-maps-scraper [options] Core Options: -input string Path to input file with queries (one per line) -results string Output file path (default: stdout) -json Output JSON instead of CSV -depth int Max scroll depth in results (default: 10) -c int Concurrency level (default: half of CPU cores) Email & Reviews: -email Extract emails from business websites -extra-reviews Collect extended reviews (up to ~300) Location Settings: -lang string Language code, e.g., 'de' for German (default: "en") -geo string Coordinates for search, e.g., '37.7749,-122.4194' -zoom int Zoom level 0-21 (default: 15) -radius float Search radius in meters (default: 10000) -grid-bbox string Bounding box for grid scraping, format: "minLat,minLon,maxLat,maxLon" -grid-cell float Grid cell size in km (default: 1.0, used with -grid-bbox) Web Server: -web Run web server mode -addr string Server address (default: ":8080") -data-folder Data folder for web runner (default: "webdata") Database: -dsn string PostgreSQL connection string -produce Produce seed jobs only (requires -dsn) Proxy: -proxies string Comma-separated proxy list Format: protocol://user:pass@host:port Export: -leadsdb-api-key Export directly to LeadsDB (get key at getleadsdb.com) Advanced: -exit-on-inactivity duration Exit after inactivity (e.g., '5m') -fast-mode Quick mode with reduced data -debug Show browser window -writer string Custom writer plugin (format: 'dir:pluginName') -browser-pool-size int Number of browser processes to launch (default: 0, derived from -c and -pages-per-browser) -pages-per-browser int Max concurrent pages per browser process (default: 1) Notes: -grid-bbox requires a valid zoom level (1-21) -fast-mode cannot be used together with -grid-bbox ``` -------------------------------- ### Export to LeadsDB using Binary Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Execute the scraper binary to export data directly to LeadsDB. Provide input queries and API key. ```bash ./google-maps-scraper \ -input queries.txt \ -leadsdb-api-key "your-api-key" \ -exit-on-inactivity 3m ``` -------------------------------- ### Run Scrapers with PostgreSQL Provider Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Execute scraper instances connected to a PostgreSQL database. Configure concurrency, depth, and the database DSN. ```bash ./google-maps-scraper \ -c 2 \ -depth 1 \ -dsn "postgres://postgres:postgres@localhost:5432/postgres" ``` -------------------------------- ### Run Scraper with Custom Writer Plugin Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Execute the scraper, specifying a custom writer plugin by its path and name. Input queries are provided via a file. ```bash ./google-maps-scraper -writer ~/plugins:MyWriter -input queries.txt ``` -------------------------------- ### Go Scraper: Keywords from File Source: https://github.com/gosom/google-maps-scraper/blob/main/examples/examples-api/README.md Pipe keywords from a file to the Go scraper. Requires Go 1.21+. ```bash cat keywords.txt | go run go/main.go -base-url "$BASE_URL" -api-key "$API_KEY" ``` -------------------------------- ### Basic CSV Lead Scrape with Docker Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/recipes.md Perform a basic scrape of Google Maps data and save results to a CSV file. Ensure the output directory exists before running. ```bash mkdir -p gmaps-output docker run \ -v gmaps-playwright-cache:/opt \ -v "$PWD/example-queries.txt:/queries.txt:ro" \ -v "$PWD/gmaps-output:/out" \ gosom/google-maps-scraper \ -input /queries.txt \ -results /out/results.csv \ -depth 1 \ -exit-on-inactivity 3m ``` -------------------------------- ### Export to LeadsDB via Environment Variable Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Configure the LeadsDB API key using an environment variable for direct data export. ```bash export LEADSDB_API_KEY="your-api-key" ./google-maps-scraper -input queries.txt -exit-on-inactivity 3m ``` -------------------------------- ### Go Scraper: Custom Output and Parallelism Source: https://github.com/gosom/google-maps-scraper/blob/main/examples/examples-api/README.md Run the Go scraper with a custom output directory and specify the number of parallel workers. Requires Go 1.21+. ```bash go run go/main.go -base-url "$BASE_URL" -api-key "$API_KEY" -o results -w 10 "cafes in athens" ``` -------------------------------- ### Format Code with gofmt Source: https://github.com/gosom/google-maps-scraper/blob/main/AGENTS.md Automatically format the project's Go code using gofmt. This ensures consistent code style, including indentation and spacing, across the entire codebase. ```bash make format ``` -------------------------------- ### Docker Mount Notes for Output Directory Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/recipes.md Mounting an output directory is more reliable than mounting a specific results file. Docker can create the directory if it doesn't exist. ```bash -v "$PWD/gmaps-output:/out" ``` -------------------------------- ### Run go vet Static Analysis Source: https://github.com/gosom/google-maps-scraper/blob/main/AGENTS.md Execute the go vet static analysis tool to check for suspicious constructs in the Go code. This command helps in finding common programming errors. ```bash make vet ``` -------------------------------- ### Initialize Cloud Provider Data Source: https://github.com/gosom/google-maps-scraper/blob/main/admin/templates/workers.html JavaScript variables to hold region and size data for DigitalOcean and Hetzner Cloud. This data is used to populate dropdowns for provisioning workers. ```javascript var doRegions = [ {{range .DORegions}} slug: "{{.Slug}}",name:"{{.Name}}" {{end}} ]; var doSizes = [ {{range .DOSizes}} slug: "{{.Slug}}",desc:"{{.Description}}",vcpus:{{.VCPUs}},mem:{{.Memory}},disk:{{.Disk}},price:{{printf "%.0f" .PriceMonthly}},regions:[] {{end}} ]; var hetznerRegions = [ {{range .HetznerRegions}} slug: "{{.Slug}}",name:"{{.Name}}" {{end}} ]; var hetznerSizes = [ {{range .HetznerSizes}} slug: "{{.Slug}}",desc:"{{.Description}}",vcpus:{{.VCPUs}},mem:{{.Memory}},disk:{{.Disk}},price:{{printf "%.0f" .PriceMonthly}},regions:[ {{range .Regions}}"{{.}} " {{end}} ] {{end}} ]; ``` -------------------------------- ### Create SSH Key Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/development-saas.md Generates a new SSH key pair for secure access to the virtual machine. Use a descriptive comment for identification. ```bash ssh-keygen -t ed25519 -C "maps-test-key" ``` -------------------------------- ### Run Tests for a Specific Package Source: https://github.com/gosom/google-maps-scraper/blob/main/AGENTS.md Execute unit tests for a particular package within the project. Replace './path/to/package' with the actual path to the package you want to test. ```bash go test ./path/to/package ``` -------------------------------- ### Run Docker Command Line Scraper Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md This command sets up a directory for output, mounts local directories for input queries and output results, and runs the scraper with specified depth and inactivity timeout. ```bash mkdir -p gmaps-output docker run \ -v gmaps-playwright-cache:/opt \ -v "$PWD/example-queries.txt:/queries.txt:ro" \ -v "$PWD/gmaps-output:/out" \ gosom/google-maps-scraper \ -input /queries.txt \ -results /out/results.csv \ -depth 1 \ -exit-on-inactivity 3m ``` -------------------------------- ### Python Scraper: Keywords as Arguments Source: https://github.com/gosom/google-maps-scraper/blob/main/examples/examples-api/README.md Run the Python scraper with keywords provided directly as command-line arguments. Requires Python 3.7+. ```bash python python/scrape.py --base-url "$BASE_URL" --api-key "$API_KEY" "cafes in athens" "hotels in berlin" ``` -------------------------------- ### Run Tests with Coverage Statistics Source: https://github.com/gosom/google-maps-scraper/blob/main/AGENTS.md Run unit tests and generate coverage statistics to understand which parts of the code are being tested. This helps in identifying areas that may require more test coverage. ```bash make test-cover ``` -------------------------------- ### Provision Self-Hosted SaaS Platform Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/recipes.md Provision the self-hosted SaaS edition of the Google Maps Scraper. This command downloads and executes a provisioning script. ```bash curl -fsSL https://raw.githubusercontent.com/gosom/google-maps-scraper/main/PROVISION | sh ``` -------------------------------- ### Copy Packages During Merge Source: https://github.com/gosom/google-maps-scraper/blob/main/migration-pro.md This script demonstrates copying packages from the 'maps-scraper-pro' repository to the 'google-maps-scraper' destination. It includes merging existing packages, copying new ones, and handling root constants and the new binary. ```bash PRO=/home/giorgos/Development/github.com/gosom/maps-scraper-pro DST=/home/giorgos/Development/github.com/gosom/google-maps-scraper # Merge SaaS postgres into existing postgres/ (add as a new file, same package) cp $PRO/postgres/postgres.go $DST/postgres/pool.go # New packages (no overlap — recursive copy includes embedded assets) cp -r $PRO/admin $DST/admin cp -r $PRO/api $DST/api cp -r $PRO/cli $DST/cli cp -r $PRO/cryptoext $DST/cryptoext cp -r $PRO/env $DST/env cp -r $PRO/httpext $DST/httpext cp -r $PRO/infra $DST/infra cp -r $PRO/log $DST/log cp -r $PRO/migrations $DST/migrations cp -r $PRO/ratelimit $DST/ratelimit cp -r $PRO/rqueue $DST/rqueue cp -r $PRO/scraper $DST/scraper # Root constants → saas/ subdirectory (package saas) mkdir -p $DST/saas cp $PRO/constants.go $DST/saas/constants.go # New binary (rename gmapspro → gmapssaas) mkdir -p $DST/cmd cp -r $PRO/cmd/gmapspro $DST/cmd/gmapssaas ``` -------------------------------- ### Use Fast Mode for Quick Data Collection Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Utilize fast mode for rapid data collection, returning up to 21 results per query. This mode is useful for basic fields but may lead to blocking as it is in Beta. ```bash ./google-maps-scraper \ -input queries.txt \ -results results.csv \ -fast-mode \ -zoom 15 \ -radius 5000 \ -geo '37.7749,-122.4194' ``` -------------------------------- ### Stop Local SaaS Development Environment Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/saas.md Stops the local development environment for the SaaS edition. ```bash make saas-dev-stop # Stop ``` -------------------------------- ### Build Custom Writer Plugin Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Compile a custom writer plugin for the scraper using Go's plugin build mode. Ensure the plugin tag is included. ```bash go build -buildmode=plugin -tags=plugin -o myplugin.so myplugin.go ``` -------------------------------- ### Run Docker Scraper to LeadsDB Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md This command configures the scraper to send results directly to LeadsDB instead of a CSV file, using an API key and specifying input, depth, and inactivity timeout. ```bash docker run \ -v gmaps-playwright-cache:/opt \ -v "$PWD/example-queries.txt:/queries.txt:ro" \ gosom/google-maps-scraper \ -input /queries.txt \ -depth 1 \ -leadsdb-api-key "your-api-key" \ -exit-on-inactivity 3m ``` -------------------------------- ### Run Unit Tests with Race Detection Source: https://github.com/gosom/google-maps-scraper/blob/main/AGENTS.md Execute all unit tests for the project, including race condition detection. This command is useful for ensuring code correctness and identifying potential concurrency issues. ```bash make test ``` -------------------------------- ### Update Google Maps Scraper SaaS Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/saas.md Updates the SaaS edition to the latest version using the provisioning script with the 'update' flag. This command should be run after initial deployment. ```bash curl -fsSL https://raw.githubusercontent.com/gosom/google-maps-scraper/main/PROVISION | sh -s update ``` -------------------------------- ### SSH into VM Source: https://github.com/gosom/google-maps-scraper/blob/main/docs/development-saas.md Connects to the virtual machine via SSH using the specified private key and IP address. Replace '' with the actual IP address obtained from 'multipass info'. ```bash ssh -i local/test_ssh_key ubuntu@ ``` -------------------------------- ### Create a new scraping job Source: https://github.com/gosom/google-maps-scraper/blob/main/README.md Initiates a new web scraping job. This endpoint allows users to define the parameters for a scraping task. ```APIDOC ## POST /api/v1/jobs ### Description Create a new scraping job. ### Method POST ### Endpoint /api/v1/jobs ### Parameters #### Request Body (No specific fields mentioned in the source, refer to OpenAPI documentation for details) ### Request Example (Refer to OpenAPI documentation for example) ### Response #### Success Response (200) (No specific fields mentioned in the source, refer to OpenAPI documentation for details) #### Response Example (Refer to OpenAPI documentation for example) ``` -------------------------------- ### Enable Backup Code Saving Source: https://github.com/gosom/google-maps-scraper/blob/main/admin/templates/2fa_setup.html This JavaScript enables the 'Enable 2FA' button only after the user checks the confirmation box, ensuring they have saved their backup codes. ```javascript document.getElementById('confirm').onchange = function() { document.getElementById('btn').disabled = !this.checked; }; ``` -------------------------------- ### SSH Key Permissions Source: https://github.com/gosom/google-maps-scraper/blob/main/admin/templates/workers.html Command to set the correct file permissions for the downloaded SSH private key. This is a security measure required by SSH. ```bash chmod 600 gmapspro_id_ed25519 ```