### Verify project setup with PowerShell commands

Source: https://github.com/diabolacal/ef-map/blob/main/GITHUB-COPILOT.md

Commands to install dependencies, run type checking, build the project, and start the development server. Essential for verifying the project setup and functionality.

```powershell
cd "C:\EF-Map-main\eve-frontier-map"
npm install
npm run typecheck
npm run build
npm run dev
```

--------------------------------

### Start EF-Map Development Server with PowerShell and npm

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

Installs project dependencies and launches the EF-Map development server via PowerShell. Changes to the eve-frontier-map folder, runs npm install (first time only), and starts the Vite dev server. Requires Node.js and npm.

```powershell
cd C:\\EF-Map-main\\eve-frontier-map\n# Install dependencies (first time only)\nnpm install\# Start dev server\nnpm run dev
```

--------------------------------

### Install Python Dependencies (Bash)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Creates a virtual environment and installs required Python packages. Requires Python and pip. Uses a requirements.txt file as input to install dependencies. Outputs an activated virtual environment. Limitations may include OS-specific activation commands.

```bash
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

```

--------------------------------

### Verify VULTUR Installation (PowerShell)

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

This PowerShell snippet verifies the installation by listing Python scripts and the README file within the cloned VULTUR repository. Ensure these files exist to confirm successful installation.

```powershell
ls *.py
```

```powershell
cat README.md
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Installs required Python packages including ijson for streaming JSON parsing, DuckDB for SQL queries, PyArrow for columnar data, pandas for data analysis, and tqdm for progress bars. Dependencies enable efficient processing of large EVE Frontier datasets.

```bash
pip install -r requirements.txt
```

```python
requirements.txt:
ijson==3.3.0          # Streaming JSON parser
duckdb==1.1.3         # In-process SQL database
pyarrow==18.0.0       # Columnar data (used by DuckDB)
pandas==2.2.3         # Data analysis (optional, for advanced queries)
tqdm==4.66.5          # Progress bars
```

--------------------------------

### Navigate to VULTUR Directory (PowerShell)

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

Demonstrates how to navigate to the VULTUR directory using PowerShell, either if installed to the C:\ drive or a user directory.

```powershell
cd C:\eve-frontier-tools
```

```powershell
cd $env:USERPROFILE\eve-frontier-tools
```

--------------------------------

### Find Custom EVE Frontier Installation

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

PowerShell command to search the C: drive for the 'EVE Frontier' directory, helpful when the EVE client is installed in a non-standard location.

```powershell
# Find EVE Frontier installation
Get-ChildItem C:\ -Recurse -Directory -Filter "EVE Frontier" -ErrorAction SilentlyContinue
```

--------------------------------

### Inspect Start of JSON File

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

Command to display the first 20 lines of a JSON file, useful for manually inspecting its structure and identifying potential parsing issues.

```powershell
# Open first few lines of stellar_systems.json
Get-Content stellar_systems.json -TotalCount 20
```

--------------------------------

### End to End Example

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

An annotated example demonstrating how to configure a REST API source with pagination and incremental loading.

```APIDOC
## End to End Example

### Description
This template demonstrates the configuration of a REST API source with cursor-based pagination and incremental loading.

```python
import dlt
from dlt.sources.rest_api import rest_api_source

# Build the REST API config with cursor-based pagination
source = rest_api_source({
    "client": {
        "base_url": "https://api.pipedrive.com/",  # Extract this from the docs/legacy code
        "auth": {
            "type": "api_key",                    # Use the documented auth type
            "name": "api_token",
            "api_key": dlt.secrets["api_token"],    # Replace with secure token reference
            "location": "query"                     # Typically a query parameter for API keys
        }
    },
    "resource_defaults": {
        "primary_key": "id",                        # Default primary key for resources
        "write_disposition": "merge",               # Default write mode
        "endpoint": {
            "params": {
                "limit": 50                         # Default query parameter for pagination size
            }
        }
    },
    "resources": [
        {
            "name": "deals",                        # Example resource name extracted from code or docs
            "endpoint": {
                "path": "v1/recents",               # Endpoint path to be appended to base_url
                "method": "GET",                    # HTTP method (default is GET)
                "params": {
                    "items": "deal"
                    "since_timestamp": "{incremental.start_value}"
                },
                "data_selector": "data.*",          # JSONPath to extract the actual data
                "paginator": {                      # Endpoint-specific paginator
                    "type": "offset",
                    "offset": 0,
                    "limit": 100
                },
                "incremental": {                    # Optional incremental configuration
                    "cursor_path": "update_time",
                    "initial_value": "2023-01-01 00:00:00"
                }
            }
        }
    ]
})

if __name__ == "__main__":
    pipeline = dlt.pipeline(
        pipeline_name="pipedrive_rest",
        destination="duckdb",
        dataset_name="pipedrive_data"
    )
    pipeline.run(source)
```

```

--------------------------------

### Start Interactive SQL Querying (Bash)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Launches the interactive SQL query tool for NDJSON data. Requires DuckDB and NDJSON files. Allows user input of SQL queries and outputs results to the console. Limitations include dependence on data availability and DuckDB syntax.

```bash
python query_data.py

```

--------------------------------

### Sample SQL Queries for EVE Data (SQL)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Example queries to extract and analyze EVE Frontier game data. Uses DuckDB on NDJSON files. Queries ship types, solar systems, and blueprints, outputting tabular results. Effective for exploration but requires accurate file paths and column knowledge.

```sql
-- List all ship types
SELECT typeID, typeName, groupID, mass, volume 
FROM 'types.ndjson' 
WHERE groupID IN (SELECT groupID FROM 'groups.ndjson' WHERE categoryID = 6)
LIMIT 10;

```

```sql
-- Find solar systems with most stargates
SELECT fromSolarSystemID, COUNT(*) as gate_count
FROM 'jumps.ndjson'
GROUP BY fromSolarSystemID
ORDER BY gate_count DESC
LIMIT 10;

```

```sql
-- Get all blueprints for a specific item
SELECT * FROM 'blueprints.ndjson' WHERE productTypeID = 45031;

```

--------------------------------

### Extract EVE Frontier Game Data (Bash)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Executes the data extraction from the EVE Frontier game client. Requires Python 3.12+ and the game installation path. Processes game files and outputs approximately 2GB of JSON data to phobos_output. Limitations involve game client being installed and accessible at the specified path.

```bash
cd tools/phobos
py -3.12 run.py --path "C:/Program Files/EVE Frontier"

```

--------------------------------

### Check EVE Client Installation Path

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

PowerShell command to verify if the EVE Frontier client is installed at the specified path. Useful for troubleshooting VULTUR extraction issues.

```powershell
Test-Path "C:\CCP\EVE Frontier"
```

--------------------------------

### Start PostgreSQL Docker Container

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/tests/postgres/README.md

Command to start the PostgreSQL instance using Docker Compose in detached mode. Builds the container if necessary and runs it in the background.

```bash
docker-compose up --build -d
```

--------------------------------

### Clone Phobos Extraction Tool (Bash)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Clones the Phobos repository for data extraction. Requires Git installed. Takes a branch-specific Git URL as input and outputs the cloned repository to tools/phobos. Limitations include dependency on specific branch availability.

```bash
git clone -b fsdbinary-t1 https://github.com/ProtoDroidBot/Phobos.git tools/phobos

```

--------------------------------

### Full dlt REST API Source Setup Example

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/codex/AGENT.md

Build a complete dlt source for a REST API with authentication, resource defaults, and endpoint-specific pagination/incremental configs. Uses rest_api_source to define client, defaults, and resources like deals with offset pagination. Run via dlt.pipeline for data loading to destinations like DuckDB.

```python
import dlt
from dlt.sources.rest_api import rest_api_source

# Build the REST API config with cursor-based pagination
source = rest_api_source({
    "client": {
        "base_url": "https://api.pipedrive.com/",  # Extract this from the docs/legacy code
        "auth": {
            "type": "api_key",                    # Use the documented auth type
            "name": "api_token",
            "api_key": dlt.secrets["api_token"],    # Replace with secure token reference
            "location": "query"                     # Typically a query parameter for API keys
        }
    },
    "resource_defaults": {
        "primary_key": "id",                        # Default primary key for resources
        "write_disposition": "merge",               # Default write mode
        "endpoint": {
            "params": {
                "limit": 50                         # Default query parameter for pagination size
            }
        }
    },
    "resources": [
        {
            "name": "deals",                        # Example resource name extracted from code or docs
            "endpoint": {
                "path": "v1/recents",               # Endpoint path to be appended to base_url
                "method": "GET",                    # HTTP method (default is GET)
                "params": {
                    "items": "deal"
                    "since_timestamp": "{incremental.start_value}"
                },
                "data_selector": "data.*",          # JSONPath to extract the actual data
                "paginator": {                      # Endpoint-specific paginator
                    "type": "offset",
                    "offset": 0,
                    "limit": 100
                },
                "incremental": {                    # Optional incremental configuration
                    "cursor_path": "update_time",
                    "initial_value": "2023-01-01 00:00:00"
                }
            }
        }
    ]
})

if __name__ == "__main__":
    pipeline = dlt.pipeline(
        pipeline_name="pipedrive_rest",
        destination="duckdb",
        dataset_name="pipedrive_data"
    )
    pipeline.run(source)
```

--------------------------------

### Install and Execute Replication Pipeline in Bash

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/sources/pg_replication/README.md

Installs dependencies from requirements.txt, runs the replication pipeline script, and displays pipeline status for verification. Assumes pip and Python environment are available. Loads data into the configured destination; use show command post-execution to confirm data processed correctly.

```bash
pip install -r requirements.txt
```

```bash
python pg_replication_pipeline.py
```

```bash
dlt pipeline pg_replication_pipeline show
```

--------------------------------

### Run Mux Pipeline - Bash

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/mux/README.md

This Bash command executes the Mux pipeline script using Python 3. It requires configured credentials and installed dependencies. No direct inputs; it loads Mux data to the destination. Outputs data to the chosen destination; may fail if credentials or setup are incomplete.

```bash
python3 mux_pipeline.py
```

--------------------------------

### Configuration Setup for Scraping Source

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/scraping/README.md

TOML configuration file showing all available settings for the scraping source. Includes batch size, queue size, timeout settings, and options for specifying start URLs either directly or via file. The system merges and deduplicates URLs when both methods are used.

```toml
[sources.scraping]
# Batch size - how many scraped results to collect
# before dispatching to DLT pipeline
batch_size = 100
# Defaul queue size
queue_size = 3000
# How log to wait before exiting
queue_result_timeout = 3.0
start_urls = [
    "https://quotes.toscrape.com/page/1/"
]
start_urls_file="/path/to/urls.txt"
```

--------------------------------

### Run the Shopify DLT pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/shopify_dlt/README.md

Executes the Shopify data pipeline script to start loading data from Shopify to the configured destination.

```bash
python3 shopify_dlt_pipeline.py
```

--------------------------------

### Initialize Mux DLT Source - Bash

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/mux/README.md

This Bash command initializes a new DLT project for the Mux verified source targeted at BigQuery. It requires the DLT CLI to be installed. No additional inputs are needed; it outputs a project directory structure. Limitations include dependency on DLT and valid destination choices.

```bash
dlt init mux bigquery
```

--------------------------------

### GET /posts

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

Retrieves a list of posts from the example API. The response includes pagination metadata with a `next` URL, allowing the dlt `json_link` paginator to automatically fetch subsequent pages.

```APIDOC
## GET /posts

### Description
Retrieves a list of posts. Supports pagination via a `next` URL embedded in the response body.

### Method
GET

### Endpoint
https://api.example.com/posts

### Parameters
#### Path Parameters
*None*

#### Query Parameters
*None*

#### Request Body
*None*

### Request Example
{ }

### Response
#### Success Response (200)
- **data** (array) - Array of post objects
- **pagination.next** (string) - URL to the next page of results

#### Response Example
{
    "data": [
        {"id": 1, "title": "Post 1"},
        {"id": 2, "title": "Post 2"},
        {"id": 3, "title": "Post 3"}
    ],
    "pagination": {
        "next": "https://api.example.com/posts?page=2"
    }
}
```

--------------------------------

### Verify JSON Files in Project Root

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

Command to change directory to the project root and list the stellar JSON files, confirming they have been copied correctly.

```powershell
cd C:\EF-Map-main
ls stellar*.json
```

--------------------------------

### Verify Python Version for Extraction

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Commands to verify Python 3.12 installation and run Phobos extraction with specific Python version. Phobos extraction tool requires Python 3.12 specifically, while query tools work with Python 3.8+.

```bash
py -3.12 --version  # Verify Python 3.12 installed
py -3.12 run.py --path "C:/Program Files/EVE Frontier"
```

--------------------------------

### Install Dependencies and Run Mux Pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/sources/mux/README.md

These bash commands install required Python packages from requirements.txt, execute the pipeline to load Mux data, and verify the output using dlt show. Requires Python 3 and pip; inputs are from configured credentials, outputs loaded data in the destination. Limitations include dependency on Mux API rate limits.

```bash
pip install -r requirements.txt
```

```bash
python3 mux_pipeline.py
```

```bash
dlt pipeline <pipeline_name> show
```

--------------------------------

### Clone EF-Map Repository (Bash)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Clones the EF-Map repository containing data query tools. Requires Git. Accepts a Git URL and outputs the repository to the local directory. No specific limitations beyond network access.

```bash
cd ../..
git clone https://github.com/Diabolacal/EF-Map.git
cd EF-Map/tools/data-query

```

--------------------------------

### Install Dependencies and Execute Salesforce Pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/salesforce/README.md

Commands to install required Python dependencies and run the Salesforce pipeline. Includes pip installation, pipeline execution, and verification commands. The pipeline name can be customized or defaults to 'salesforce'.

```bash
pip install -r requirements.txt
```

```bash
python3 salesforce_pipeline.py
```

```bash
dlt pipeline <pipeline_name> show
```

--------------------------------

### Install dlt and Dependencies (Shell)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/sources/unstructured_data/README.md

Installs the dlt library with DuckDB destination and requirements from requirements.txt. Requires Python 3.x. Outputs installed packages ready for pipeline initialization. No limitations noted beyond Python version.

```shell
pip install dlt[duckdb]
```

```shell
pip install -r requirements.txt
```

--------------------------------

### Run Scraping Pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/scraping/README.md

Shell commands to install project dependencies and execute the scraping pipeline. First installs required packages from requirements.txt, then runs the main scraping pipeline script that contains the spider implementation and extraction logic.

```shell
pip install -r requirements.txt
python scraping_pipeline.py
```

--------------------------------

### Run Matomo Pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/matomo/README.md

Execute the Python script to load Matomo data into the destination. The pipeline processes endpoints like visits and reports based on config. Outputs data tables verifiable via DLT commands.

```bash
python matomo_pipeline.py
```

--------------------------------

### Query Manufacturing Data

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Demonstrates SQL queries for accessing blueprint and industry activity information. Includes queries for listing blueprints with production limits and manufacturing time requirements from the industry activities dataset.

```sql
-- List all blueprints
SELECT blueprintTypeID, productTypeID, maxProductionLimit 
FROM 'blueprints.ndjson';

-- Manufacturing time
SELECT blueprintTypeID, time 
FROM 'industry_activities.ndjson' 
WHERE activityID = 1  -- Manufacturing
ORDER BY time DESC;
```

--------------------------------

### Running Example Script (Python)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/CONTRIBUTING.md

Shows how to run the example script after making changes.

```shell
python my_source_pipeline.py
```

--------------------------------

### Generate Data Catalog (Bash)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Creates a detailed catalog of NDJSON files with samples. Requires Python and data files. Outputs a markdown file (4,558 lines) summarizing structures. Already generated; re-running updates it without specific limitations.

```bash
python generate_catalog.py

```

--------------------------------

###  Python SQLite module availability

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

Executes a short Python command to import the sqlite3 module and print a confirmation. Detects whether the Python installation was built with SQLite support, a prerequisite for EF‑Map scripts.

```powershell
python -c "import sqlite3; print('SQLite OK')"
```

--------------------------------

### Implement Analytics Event Tracking in React

Source: https://github.com/diabolacal/ef-map/blob/main/docs/GOOGLE_ANALYTICS_SETUP.md

TypeScript examples showing how to import and use the analytics utility functions for tracking custom events. Demonstrates route calculation tracking with parameters and cinematic mode activation tracking.

```typescript
import { trackEvent } from './utils/analytics';

// Example: Track route calculations
trackEvent('route_calculated', {
  from_system: 'Jita',
  to_system: 'Amarr',
  waypoints: 5,
  algorithm: 'A*'
});

// Example: Track cinematic mode usage
trackEvent('cinematic_enter');
```

--------------------------------

### Specify EVE Path for VULTUR Extraction

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

Command to manually specify the EVE client installation path when VULTUR fails to auto-detect it. This is a common solution for 'VULTUR extraction failed: EVE Frontier not found' errors.

```powershell
python export_static_data.py --eve-path "C:\CCP\EVE Frontier"
```

--------------------------------

### Execute Database Verification Script via PowerShell

Source: https://github.com/diabolacal/ef-map/blob/main/docs/VULTUR_SETUP_GUIDE.md

Runs the verification script from the EF-Map-main directory using PowerShell. Executes the Python script and displays validation results. Ensure Python is installed and the script path is correct.

```powershell
cd C:\\EF-Map-main\npython verify_map_database.py
```

--------------------------------

### Initialize Matomo Pipeline with DLT

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/matomo/README.md

This command initializes a new data pipeline project using the Matomo verified source and DuckDB as the destination. It sets up the basic structure including necessary files. Alternatives include other destinations like Redshift or BigQuery.

```bash
dlt init matomo duckdb
```

--------------------------------

### Configure Base OS on VPS

Source: https://github.com/diabolacal/ef-map/blob/main/docs/hetzner-cloud-migration-playbook.md

Updates the OS, installs necessary packages (Docker, Git, Fail2ban, UFW), and adds the current user to the Docker group for permission management.

```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y docker.io docker-compose-plugin git fail2ban ufw
sudo usermod -aG docker $USER
```

--------------------------------

### Initialize Postgres Replication Pipeline in Bash

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/sources/pg_replication/README.md

Initializes a new dlt pipeline for Postgres replication using DuckDB as the default destination. Requires the dlt CLI tool to be installed. Outputs a project directory with pipeline configuration; users can select alternative destinations if needed.

```bash
dlt init pg_replication duckdb
```

--------------------------------

### Initialize Zendesk pipeline with BigQuery

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/zendesk/README.md

Command to initialize a new Zendesk pipeline with BigQuery as the destination. Other supported destinations include Redshift and DuckDB.

```bash
dlt init zendesk bigquery
```

--------------------------------

### Install required Python dependencies for inbox pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/inbox/README.md

Installs the project dependencies listed in requirements.txt and additional packages for Gmail access and PDF parsing. Run before executing the pipeline.

```bash
pip install -r requirements.txt
# For Gmail:
pip install google-api-python-client>=2.86.0
pip install google-auth-oauthlib>=1.0.0
pip install google-auth-httplib2>=0.1.0
# For PDF parsing:
pip install PyPDF2
```

--------------------------------

### Linting and Formatting Code (Shell)

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/CONTRIBUTING.md

Shows how to run the linter and formatter using a Makefile.

```shell
make lint-code
```

--------------------------------

### Provision Server and Volume in Hetzner

Source: https://github.com/diabolacal/ef-map/blob/main/docs/hetzner-cloud-migration-playbook.md

Creates a Hetzner server with specified configuration and attaches a data volume. Includes server type, image, location, and firewall settings.

```powershell
hcloud server create --name ef-map-vps --type cx22 --image ubuntu-22.04 --location fsn1 --ssh-key ef-map-ci --firewall ef-map-vps
hcloud volume create --name ef-map-data --size 80 --server ef-map-vps --format ext4
```

--------------------------------

### GET /activities/stream

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

Retrieves recent activities using cursor-based pagination.

```APIDOC
## GET /activities/stream

### Description
Fetches recent activities. Uses cursor-based pagination.

### Method
GET

### Endpoint
/activities/stream

### Parameters
#### Path Parameters
*None*

#### Query Parameters
- **page_size** (integer) - Optional - Number of activities to return. Default: 100.
- **next_page_cursor** (string) - Optional - Cursor from the previous response's `meta.next_page` field.

### Request Example
{
  "page_size": 100,
  "next_page_cursor": "aabbccddeeff"
}

### Response
#### Success Response (200)
- **activities** (array) - List of activity objects.
- **meta** (object) - Pagination metadata.

### Response Example
{
  "activities": [
    {
      "id": "act1",
      "...": "..."
    }
  ],
  "meta": {
    "next_page": "aabbccddeeff"
  }
}
```

--------------------------------

### GET /v2/orders

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

Retrieves a paginated list of orders with optional status filtering.

```APIDOC
## GET /v2/orders

### Description
Fetches a list of orders. Supports offset-based pagination and optional status filtering.

### Method
GET

### Endpoint
/v2/orders

### Parameters
#### Path Parameters
*None*

#### Query Parameters
- **limit** (integer) - Optional - Max items per page. Default: 50.
- **offset** (integer) - Optional - Number of items to skip. Default: 0.
- **status** (string) - Optional - Filter by order status (e.g., 'completed', 'pending').

### Request Example
{
  "limit": 50,
  "offset": 0,
  "status": "completed"
}

### Response
#### Success Response (200)
- **orders** (array) - List of order objects.
- **total** (integer) - Total number of orders matching the filter.

### Response Example
{
  "orders": [
    {
      "id": "123",
      "status": "completed",
      "...": "..."
    }
  ],
  "total": 200
}
```

--------------------------------

### POST /session/start-session

Source: https://github.com/diabolacal/ef-map/blob/main/docs/initiatives/GAME_OVERLAY_PLAN.md

Starts a new tracking session, creating a timestamped file for per-session visited systems data.

```APIDOC
## POST /session/start-session

### Description
Initiates a new session for tracking visited systems, generating a unique session ID and starting the count.

### Method
POST

### Endpoint
/session/start-session

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```
POST /session/start-session
```

### Response
#### Success Response (200)
- **session_id** (string) - The ID of the newly created session
- **timestamp** (string) - Start time of the session

#### Response Example
```
{
  "session_id": "session_123",
  "timestamp": "2025-10-28T10:00:00Z"
}
```
```

--------------------------------

### Query Solar Systems and Navigation

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Shows SQL queries for exploring EVE Frontier solar system geography and stargate connections. Includes queries for listing regions, finding systems within regions, and examining stargate jump connections using NDJSON data files.

```sql
-- List all regions
SELECT regionID, regionName FROM 'regions.ndjson';

-- Systems in a region
SELECT solarSystemID, solarSystemName, security 
FROM 'systems.ndjson' 
WHERE regionID = 10000001;

-- Stargate connections for a system
SELECT * FROM 'jumps.ndjson' 
WHERE fromSolarSystemID = 30000001;
```

--------------------------------

### Fetch Activity Stream - GET

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

This endpoint retrieves recent activities using cursor-based pagination. It supports specifying a page size and providing a cursor from a previous response. The endpoint uses a GET request.

```http
GET /activities/stream

Query Parameters:
- page_size (integer, optional, default: 100)
- next_page_cursor (string, optional)

Response:
{
  "activities": [...],
  "meta": {
    "next_page": "aabbccddeeff"
  }
}
```

--------------------------------

### Fetch List Orders - GET

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

This endpoint retrieves a list of orders using offset-based pagination. It allows filtering by status and limiting the number of items per page. The endpoint is accessed via a GET request.

```http
GET /v2/orders

Query Parameters:
- limit (integer, optional, default: 50)
- offset (integer, optional, default: 0)
- status (string, optional)
```

--------------------------------

### Initialize Shopify DLT pipeline with BigQuery

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/shopify_dlt/README.md

Initializes a new Shopify data pipeline project with BigQuery as the destination. Other supported destinations include Redshift and DuckDB.

```bash
dlt init shopify_dlt bigquery
```

--------------------------------

### Display Project File Structure

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Shows the complete directory structure of the EF-Map project, including extraction tools, extracted data, and query tools. The structure indicates data flow from Phobos extraction through JSON/NDJSON conversion to SQL querying.

```bash
tools/
├── phobos/                          # Extraction tool (clone separately)
│   └── run.py                       # Extractor script
├── phobos_output/                   # Extracted data (gitignored)
│   ├── *.json                       # Raw JSON (121 files, ~2GB)
│   └── ndjson/                      # Converted NDJSON (120 files, 888MB)
│       ├── systems.ndjson
│       ├── types.ndjson
│       └── ...
└── data-query/                      # Query tools (this directory)
    ├── PHOBOS_DATA_CATALOG.md       # Complete data catalog
    ├── QUICKSTART.md                # This file
    ├── README.md                    # Detailed documentation
    ├── requirements.txt             # Python dependencies
    ├── json_to_ndjson.py            # NDJSON converter (single file)
    ├── convert_all.py               # Batch converter
    ├── query_data.py                # SQL query interface
    ├── view_full_record.py          # Record viewer
    └── generate_catalog.py          # Catalog generator
```

--------------------------------

### Incremental Loading Configuration

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

Describes how to configure incremental loading for getting only new data from endpoints.

```APIDOC
## Incremental Loading Configuration

### Description
Configure incremental loading to efficiently extract only new data from endpoints. Identify query parameters to filter for newer data.

### Example
```py
{
    "path": "posts",
    "data_selector": "results",
    "params": {
        "created_since": "{incremental.start_value}",  # Uses cursor value in query parameter
    },
    "incremental": {
        "cursor_path": "created_at",
        "initial_value": "2024-01-25T00:00:00Z",
    },
}
```

```

--------------------------------

### Query Ships and Items Data

Source: https://github.com/diabolacal/ef-map/blob/main/tools/data-query/QUICKSTART.md

Demonstrates SQL queries for accessing ship type information and attributes. Includes queries for ship types with physical properties and joins between types and dogma attributes. Uses DuckDB to query NDJSON files containing EVE Frontier ship data.

```sql
-- Get all ship types with mass/volume
SELECT typeID, typeName, mass, volume, capacity 
FROM 'types.ndjson' 
WHERE categoryID = 6
LIMIT 20;

-- Find ship attributes
SELECT t.typeID, t.typeName, td.attributeID, td.value
FROM 'types.ndjson' t
JOIN 'typedogma.ndjson' td ON t.typeID = td.typeID
WHERE t.groupID = 25  -- Frigates
LIMIT 50;
```

--------------------------------

### GET /v2/orders

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/codex/AGENT.md

Fetches a list of orders with offset-based pagination. Allows filtering by status.

```APIDOC
## GET /v2/orders

### Description
Fetches a list of orders. This endpoint uses offset-based pagination.

### Method
GET

### Endpoint
/v2/orders

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Max items per page. Default: 50.
- **offset** (integer) - Optional - Number of items to skip. Default: 0.
- **status** (string) - Optional - Filter by status (e.g., 'completed', 'pending').

### Response
#### Success Response (200)
- **orders** (array) - List of order objects.

#### Response Example
{
  "orders": [
    {
      "id": "order123",
      "status": "completed"
    }
  ]
}
```

--------------------------------

### Install Dependencies and Execute Pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/google_analytics/README.md

Installs Python requirements from requirements.txt and runs the google_analytics_pipelines.py script to load data. The pipeline name is dlt_google_analytics_pipeline by default. Inputs include configured credentials; outputs loaded tables for metrics and dimensions.

```bash
pip install -r requirements.txt
```

```bash
python google_analytics_pipelines.py
```

--------------------------------

### GET /activities/stream

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/codex/AGENT.md

Fetches recent activities with cursor-based pagination. Supports custom page sizes.

```APIDOC
## GET /activities/stream

### Description
Fetches recent activities. Uses cursor-based pagination.

### Method
GET

### Endpoint
/activities/stream

### Parameters
#### Query Parameters
- **page_size** (integer) - Optional - Number of activities. Default: 100.
- **next_page_cursor** (string) - Optional - Cursor from the previous response's `meta.next_page` field.

### Response
#### Success Response (200)
- **activities** (array) - List of activity objects.
- **meta.next_page** (string) - Cursor for the next page of results.

#### Response Example
{
  "activities": [
    {
      "id": "activity1",
      "type": "login"
    }
  ],
  "meta": {
    "next_page": "aabbccddeeff"
  }
}
```

--------------------------------

### POST /bookmarks/create

Source: https://github.com/diabolacal/ef-map/blob/main/docs/initiatives/GAME_OVERLAY_PLAN.md

Creates a new bookmark for a system, with optional notes and tribe placement when the user is authenticated.

```APIDOC
## POST /bookmarks/create\n\n### Description\nCreates a new bookmark for the specified system. Supports optional notes and the ability to add the bookmark to a tribe folder when the user is authenticated and belongs to a player‑created tribe.\n\n### Method\nPOST\n\n### Endpoint\n/bookmarks/create\n\n### Parameters\n#### Request Body\n- **system_id** (integer) - Required - Identifier of the system to bookmark.\n- **system_name** (string) - Required - Human‑readable name of the system.\n- **notes** (string) - Optional - User‑provided notes for the bookmark.\n- **for_tribe** (boolean) - Optional - When true, the bookmark is stored in the user's tribe folder.\n\n### Request Example\n{\n  "system_id": 12345,\n  "system_name": "Mahnna",\n  "notes": "Strategic location for resource gathering",\n  "for_tribe": true\n}\n\n### Response\n#### Success Response (200)\n- **status** (string) - \"created\"\n- **bookmark_id** (integer) - Identifier of the newly created bookmark.\n\n### Response Example\n{\n  "status": "created",\n  "bookmark_id": 9876\n}
```

--------------------------------

### Install Dependencies and Run Pipeline

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/sources/shopify_dlt/README.md

Executes the complete pipeline workflow including dependency installation and pipeline execution. The commands install required Python packages and run the main pipeline script. After execution, users can verify the pipeline status using the show command.

```bash
pip install -r requirements.txt
```

```bash
python3 shopify_dlt_pipeline.py
```

```bash
dlt pipeline <pipeline_name> show
```

--------------------------------

### Initialize Notion Pipeline with dlt

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/notion/README.md

The dlt init command sets up the Notion verified source pipeline for a specified destination such as BigQuery, creating necessary files and configuration. Dependencies include dlt installation; inputs are source name and destination; outputs are pipeline structure files. Limitation: Requires pre-installed dlt.

```bash
dlt init notion bigquery
```

--------------------------------

### Common Documentation Patterns

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/amp/AGENT.md

Provides examples of common documentation patterns for API authentication and endpoint descriptions, illustrating how to format these sections for clarity.

```APIDOC
## Authentication
To authenticate, include your API key in the `Authorization: Bearer <your_token>` header.

## Endpoint Documentation Example (with variations)
```

--------------------------------

### Initialize Personio pipeline with dlt

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/verified-sources-0.5/sources/personio/README.md

Initializes the Personio data pipeline using dlt with a specified destination. The example uses DuckDB but supports other destinations like Redshift or BigQuery.

```bash
dlt init personio duckdb
```

--------------------------------

### Python Example: json_link Pagination

Source: https://github.com/diabolacal/ef-map/blob/main/tools/worldapi-pipeline/worldapi/worldapi_pipeline/rest_api/verified-sources-master/ai/codex/AGENT.md

Demonstrates configuring the `json_link` paginator in Python to extract the next page URL from a JSON response.

```python
{ "path": "posts", "paginator": { "type": "json_link", "next_url_path": "pagination.next" } }
```