### Docker Command Example Source: https://docs.everef.net/commands/scrape-market-history An example command for interacting with Docker, as listed under the project's commands. ```bash docker run --rm -v $(pwd):/app your-eve-ref-image docker-command-here ``` -------------------------------- ### Run EVE Ref API Locally Source: https://docs.everef.net/api This command allows you to run the EVE Ref API locally using Docker. It pulls the latest image and starts the API service. ```docker docker run -it --rm autonomouslogic/eve-ref:latest api ``` -------------------------------- ### Sync Market Orders with rclone on Docker Source: https://docs.everef.net/datasets/downloading-datasets This command demonstrates how to sync market order history data using rclone within a Docker container. It mounts the current directory to persist downloaded data. Requires Docker to be installed. ```shell docker run --rm -v $(pwd):/data -w /data rclone/rclone sync -v --checkers 2 --transfers 2 --multi-thread-streams 0 --http-url https://data.everef.net :http:/market-orders/history/2023 pathtosyncto ``` -------------------------------- ### Sync Market Orders with rclone Source: https://docs.everef.net/datasets/downloading-datasets This command uses rclone to synchronize market order history data for a specific year. It leverages multiple concurrent downloads for efficiency. Ensure rclone is installed and configured. ```shell rclone sync -v --checkers 2 --transfers 2 --multi-thread-streams 0 --http-url https://data.everef.net :http:/market-orders/history/2023 pathtosync ``` -------------------------------- ### Mount Data Directory with rclone Source: https://docs.everef.net/datasets/downloading-datasets This command mounts the entire data directory from data.everef.net onto the local filesystem using rclone. It configures caching and timeouts for optimal performance. Useful for accessing data as if it were local. ```shell rclone mount --attr-timeout 1m --dir-cache-time 1m --vfs-cache-mode full --checkers 2 --transfers 2 --multi-thread-streams 0 --http-url https://data.everef.net :http:/ pathtomount ``` -------------------------------- ### Download Market Orders History using Docker Source: https://docs.everef.net/datasets/downloading-datasets This snippet demonstrates how to use the `mwendler/wget` Docker image to download market order history data from data.everef.net. It mounts the current directory to `/data` inside the container, sets the working directory, and uses `wget` with specific flags to recursively download files, excluding `index.html`, and disabling certificate checks. ```shell docker run --rm -v $(pwd):/data -w /data mwendler/wget -r -np -N -nv --domains=data.everef.net -R index.html --no-check-certificate https://data.everef.net/market-orders/history/2023/ ``` -------------------------------- ### Fetch Market Structures Source: https://docs.everef.net/datasets/structures This command fetches the latest market structure data from Everef.net and filters for structures that are marked as market structures using jq. It requires curl and jq to be installed. ```bash curl -s https://data.everef.net/structures/structures-latest.v2.json | jq '.[] | select(.is_market_structure)' ``` -------------------------------- ### Download Market Orders with wget Source: https://docs.everef.net/datasets/downloading-datasets This command uses wget to recursively download market order data for a specific year. It includes options to prevent ascending directories, enforce timestamp checking, and restrict downloads to the specified domain. Note that wget does not support multiple concurrent downloads. ```shell wget -r -np -N -nv --domains=data.everef.net -R index.html https://data.everef.net/market-orders/history/2023/ ``` -------------------------------- ### Configure Initial Market History Import Date Source: https://docs.everef.net/commands/import-market-history Set the environment variable IMPORT_MARKET_HISTORY_MIN_DATE to '2003-01-01' to import all market history data starting from January 1, 2003. This variable can be omitted to use the default behavior. ```shell IMPORT_MARKET_HISTORY_MIN_DATE=2003-01-01 ``` -------------------------------- ### Invention Cost Calculation Response Example Source: https://docs.everef.net/api/industry-cost This JSON object represents a typical response from the invention cost API endpoint. It provides detailed breakdown of costs, materials, probabilities, and time estimations for an invention job. ```JSON { "invention" : { "2622": { "product_id": 2622, "runs": 10, "time": "PT136H40M23S", "materials": { "20418": { "type_id": 20418, "quantity": 10, "cost": 1043000.50 }, "20420": { "type_id": 20420, "quantity": 10, "cost": 1020634.60 }, "34206": { "type_id": 34206, "quantity": 10, "cost": 4285722.00 } }, "estimated_item_value": 16852252, "system_cost_index": 39468, "system_cost_bonuses": -1184, "facility_tax": 1685, "scc_surcharge": 13482, "alpha_clone_tax": 0, "total_job_cost": 53451, "total_material_cost": 6349357.10, "total_cost": 6402808.10, "blueprint_id": 805, "probability": 0.49583333333333335, "runs_per_copy": 12, "units_per_run": 5000, "expected_copies": 4.958333333333334, "expected_runs": 59.50000000000001, "expected_units": 297500.00000000006, "me": 3, "te": 12, "job_cost_base": 337045, "avg_time_per_copy": "PT27H33M51.529S", "avg_time_per_run": "PT2H17M49.294S", "avg_time_per_unit": "PT1.653S", "avg_cost_per_copy": 1291322.64, "avg_cost_per_run": 107610.22, "avg_cost_per_unit": 21.52 } } } ``` -------------------------------- ### EVE Ref REST API Endpoints Source: https://docs.everef.net/datasets/reference-data Provides access to various EVE Online game data resources. The full OpenAPI specification is available on GitHub. Examples include fetching lists of resources or specific items by ID. ```APIDOC OpenAPI Specification: URL: https://github.com/autonomouslogic/eve-ref/blob/main/spec/reference-data.yaml Resource Paths: - /blueprints Description: Retrieves a list of all blueprints. Example: https://ref-data.everef.net/blueprints - /blueprints/{id} Description: Retrieves a specific blueprint by its ID. Example: https://ref-data.everef.net/blueprints/999 - /categories Description: Retrieves a list of all item categories. Example: https://ref-data.everef.net/categories - /categories/{id} Description: Retrieves a specific item category by its ID. Example: https://ref-data.everef.net/categories/4 - /dogma_attributes Description: Retrieves a list of all Dogma attributes. Example: https://ref-data.everef.net/dogma_attributes - /dogma_attributes/{id} Description: Retrieves a specific Dogma attribute by its ID. Example: https://ref-data.everef.net/dogma_attributes/37 - /dogma_effects Description: Retrieves a list of all Dogma effects. Example: https://ref-data.everef.net/dogma_effects - /dogma_effects/{id} Description: Retrieves a specific Dogma effect by its ID. Example: https://ref-data.everef.net/dogma_effects/11 - /groups Description: Retrieves a list of all item groups. Example: https://ref-data.everef.net/groups - /groups/{id} Description: Retrieves a specific item group by its ID. Example: https://ref-data.everef.net/groups/18 - /market_groups Description: Retrieves a list of all market groups. Example: https://ref-data.everef.net/market_groups - /market_groups/{id} Description: Retrieves a specific market group by its ID. Example: https://ref-data.everef.net/market_groups/1857 - /meta_groups Description: Retrieves a list of all meta groups. Example: https://ref-data.everef.net/meta_groups - /meta_group/{id} Description: Retrieves a specific meta group by its ID. Example: https://ref-data.everef.net/meta_group/6 - /mutaplasmids Description: Retrieves a list of all mutaplasmids. Example: https://ref-data.everef.net/mutaplasmids - /mutaplasmids/{id} Description: Retrieves a specific mutaplasmid by its ID. Example: https://ref-data.everef.net/mutaplasmids/52225 - /regions Description: Retrieves a list of all regions. Example: https://ref-data.everef.net/regions - /regions/{id} Description: Retrieves a specific region by its ID. Example: https://ref-data.everef.net/regions/10000002 - /schematics Description: Retrieves a list of all schematics. Example: https://ref-data.everef.net/schematics - /schematics/{id} Description: Retrieves a specific schematic by its ID. Example: https://ref-data.everef.net/schematics/65 - /skills Description: Retrieves a list of all skills. Example: https://ref-data.everef.net/skills - /skills/{id} Description: Retrieves a specific skill by its ID. Example: https://ref-data.everef.net/skills/3336 - /types Description: Retrieves a list of all item types. Example: https://ref-data.everef.net/types - /types/{id} Description: Retrieves a specific item type by its ID. Example: https://ref-data.everef.net/types/645 - /units Description: Retrieves a list of all units. Example: https://ref-data.everef.net/units - /units/{id} Description: Retrieves a specific unit by its ID. Example: https://ref-data.everef.net/units/1 ``` -------------------------------- ### BuildRefData CLI Command and Data Loading Process Source: https://docs.everef.net/commands/build-ref-data Details the core functionality of the `BuildRefData` CLI command. It outlines the configuration via `refdata.yaml`, the roles of `SdeLoader` and `EsiLoader` in reading data, the use of MVStore for intermediate storage, and the final merging process to create the reference data file. ```APIDOC BuildRefData: Description: Builds and uploads the main reference data file. Configuration: Uses `refdata.yaml` for processing sources. Data Loading: - `SdeLoader`: Loads data from the SDE (Static Data Export). - `EsiLoader`: Loads data from the ESI (EVE Swagger Interface). - Loaders read input files and output objects. Storage: - MVStore is used to store intermediate objects. - Maps are created for each type and source (e.g., `types/SDE`, `types/ESI`). Merging: - All loader outputs are merged into the final reference data. - Loaders output objects in their final, ready-to-merge form. ``` -------------------------------- ### EVE Ref Commands Source: https://docs.everef.net/commands/scrape-market-history Lists various command-line utilities provided by the EVE Ref project for data management and processing. Includes commands for Docker operations, data building, and scraping market data. ```APIDOC ## Commands * `Docker`: Commands related to Docker operations. * `Field Replication`: Commands for handling field replication. * `Build Ref Data`: Commands to build reference data. * `Import Market History`: Commands to import market history data. * `Scrape Market History`: Commands to scrape market history data. * `Sources`: Details on data sources used for scraping. * `Additional sources on 2023-10-02`: Information on new sources added on a specific date. * `Scrape Market Orders`: Commands to scrape market order data. * `Scrape Public Contracts`: Commands to scrape public contract data. * `Sync Fuzzwork Ordersets`: Commands to synchronize Fuzzwork ordersets. ``` -------------------------------- ### EVE Ref Industry Cost API Endpoint Source: https://docs.everef.net/api/industry-cost Provides accurate industry calculations for EVE Online manufacturing and invention. Accepts GET requests and requires specific item or blueprint IDs. Designed for ease of use from any programming language. ```APIDOC Endpoint: https://api.everef.net/v1/industry/cost Method: GET Description: Calculates industry costs for EVE Online items, supporting manufacturing and invention scenarios. Parameters: - product_id (integer, required): The item ID for which to calculate manufacturing costs. If used for invention, this is the blueprint to be invented. - blueprint_id (integer, optional): The source blueprint type ID. Used to calculate all activities on a specific blueprint, or when an item can come from multiple blueprints. Functionality: - For manufacture: Supply the `product_id` parameter of an item. - For manufacture of items requiring invention: Supply the `product_id` parameter of the item; invention will be automatically calculated. - For invention only: Supply the `product_id` parameter of the blueprint to be invented. - For items from multiple blueprints: Set `blueprint_id` as the source blueprint type ID. - To calculate all activities on a specific blueprint: Set `blueprint_id`. Returns: JSON object containing detailed cost calculations, material requirements, and invention outcomes. ``` -------------------------------- ### Running the EVE Ref API Locally Source: https://docs.everef.net/api/industry-cost This command initiates the EVE Ref API container on your local machine using Docker. It pulls the latest 'latest' tag of the 'autonomouslogic/eve-ref' image, runs it interactively, and automatically removes the container once it stops. ```docker docker run -it --rm autonomouslogic/eve-ref:latest api ``` -------------------------------- ### Build EVE Ref Docker Image Source: https://docs.everef.net/commands/docker Command to build the Docker image for EVE Ref using the provided Makefile. ```makefile make docker ``` -------------------------------- ### Get Public Contract Bids by Contract ID Source: https://docs.everef.net/datasets/public-contracts Retrieves all bid entries for a given public contract ID from EVE Online. This API includes standard bid details and additional fields provided by EVE Ref for enhanced data management. ```APIDOC GET /contracts/public/bids/{contract_id}/ Description: Retrieves all bid entries for a given public contract ID. Parameters: - contract_id (integer): The ID of the contract. ESI Fields: - amount (number): The bid amount. - bid_id (integer): The ID of the bid. - date_bid (string): The date the bid was placed (ISO 8601 format). EVE Ref Added Fields: - http_last_modified (string): Timestamp of the last HTTP modification. - contract_id (integer): Identifier for the contract. ``` -------------------------------- ### Get Dogma Dynamic Item Effects by Type and Item Source: https://docs.everef.net/datasets/public-contracts Retrieves dynamic item effects for a specific item type and ID from the EVE Online Dogma system. This endpoint provides base ESI fields and additional columns added by EVE Ref for contract tracking. ```APIDOC GET /dogma/dynamic/items/{type_id}/{item_id}/ Description: Retrieves dynamic item effects for a specific item type and ID. Parameters: - type_id (integer): The ID of the item type. - item_id (integer): The ID of the specific item. ESI Fields: - effect_id (integer): The ID of the dogma effect. - is_default (boolean): Indicates if the effect is default. EVE Ref Added Fields: - contract_id (integer): Identifier for the contract. - item_id (integer): Identifier for the item. - http_last_modified (string): Timestamp of the last HTTP modification. ``` -------------------------------- ### Adding New Resources to BuildRefData Source: https://docs.everef.net/commands/build-ref-data Provides instructions on how to add new data resources to the EVE Ref project. This involves updating loaders, adding sample data, and ensuring model consistency. ```APIDOC Adding New Resources: Steps: 1. Set up resources in ESI and SDE loaders. 2. Add samples to relevant resource directories. 3. Add tests for both loaders and the main command. Sample Sources: - Input samples: Directly from ESI scrape or SDE dump. - Output samples: Constructed manually. Verification: - Add to `PublishRefData` and `PublishRefDataTest`. - Run `VerifyRefDataModels` to ensure model existence. ``` -------------------------------- ### Testing BuildRefData Source: https://docs.everef.net/commands/build-ref-data Describes the testing strategy for the `BuildRefData` command, including the structure of test resources and the classes used for validation. It covers end-to-end testing and specific loader tests. ```APIDOC BuildRefData Testing: Test Resources: - `resources/refdata/esi`: Samples of ESI files. - `resources/refdata/sde`: Samples of SDE files. - `resources/refdata/refdata`: Expected final format (loaded, transformed, merged). Testing Classes: - `BuildRefDataTest`: Performs full end-to-end testing. - `EsiLoaderTest`: Tests the `EsiLoader` with contextual resources. - `SdeLoaderTest`: Tests the `SdeLoader` with contextual resources. Intermediate Output Testing: - Samples for intermediate loader outputs are available in contextual resource folders. ``` -------------------------------- ### Everef API: Calculate Manufacturing Cost Source: https://docs.everef.net/api/industry-cost This API endpoint calculates the estimated cost for manufacturing items in the game EVE Online. It requires parameters specifying the product, manufacturing runs, efficiency levels (ME/TE), structure type, security status, installed rigs, and various cost modifiers. The response provides a detailed breakdown of material costs, job costs, and estimated item value, including potential invention steps. ```APIDOC GET https://api.everef.net/v1/industry/cost?product_id=22430&runs=8&me=4&te=4&structure_type_id=35827&security=LOW_SEC&rig_id=37180&rig_id=37183&system_cost_bonus=-0.5&manufacturing_cost=0.0129&facility_tax=0.02 Parameters: - product_id: The ID of the item to manufacture. - runs: The number of manufacturing runs. - me: The Material Efficiency level. - te: The Time Efficiency level. - structure_type_id: The ID of the structure used for manufacturing. - security: The security status of the system (e.g., LOW_SEC). - rig_id: IDs of installed rigs (can be specified multiple times). - system_cost_bonus: The system cost bonus applied. - manufacturing_cost: The system's manufacturing cost index. - facility_tax: The tax applied by the facility. Response Structure (JSON): { "manufacturing" : { "[product_id]" : { "product_id" : integer, "runs" : number, "time" : string, // ISO 8601 duration format "materials" : { "[type_id]" : { "type_id" : integer, "quantity" : number, "cost_per_unit" : number, "cost" : number } }, "estimated_item_value" : number, "system_cost_index" : number, "system_cost_bonuses" : number, "facility_tax" : number, "scc_surcharge" : number, "alpha_clone_tax" : number, "total_job_cost" : number, "total_material_cost" : number, "total_cost" : number, "blueprint_id" : integer, "units" : number, "units_per_run" : number, "time_per_run" : string, // ISO 8601 duration format "time_per_unit" : string, // ISO 8601 duration format "total_cost_per_run" : number, "total_cost_per_unit" : number } }, "invention" : { "[blueprint_id]" : { "product_id" : integer, "runs" : number, "time" : string, // ISO 8601 duration format "materials" : { "[type_id]" : { "type_id" : integer, "quantity" : number, "cost_per_unit" : number, "cost" : number } }, "estimated_item_value" : number, "system_cost_index" : number, "system_cost_bonuses" : number, "facility_tax" : number, "scc_surcharge" : number, "alpha_clone_tax" : number, "total_job_cost" : number, "total_material_cost" : number, "total_cost" : number, "blueprint_id" : integer, "probability" : number, "runs_per_copy" : number, "units_per_run" : number, "expected_copies" : number, "expected_runs" : number, "expected_units" : number, "me" : integer, "te" : integer, "job_cost_base" : number, "avg_time_per_copy" : string, // ISO 8601 duration format "avg_time_per_run" : string, // ISO 8601 duration format "avg_time_per_unit" : string, // ISO 8601 duration format "avg_cost_per_copy" : number, "avg_cost_per_run" : number, "avg_cost_per_unit" : number } } } ``` -------------------------------- ### Run EVE Ref Docker Container Source: https://docs.everef.net/commands/docker Command to execute the EVE Ref Docker image. Replace 'command' with the desired operation to perform inside the container. ```bash docker run autonomouslogic/everef:latest command ``` -------------------------------- ### Fuzzwork Market Orders API Reference Source: https://docs.everef.net/datasets/fuzzwork-ordersets Reference to the Fuzzwork Market Orders API, which provides EVE Online market data. The EVE Ref project mirrors this data. ```APIDOC API Endpoint: https://data.everef.net/fuzzwork/ordersets/ Description: This endpoint mirrors the market orders data provided by Fuzzwork. It is similar to the Market Order Snapshots dataset. Source API: https://market.fuzzwork.co.uk/api/ Data Provided: - Market order details for EVE Online. - Likely includes information such as item type, price, quantity, location, etc. Usage: Accessing the data.everef.net URL provides a mirror of the Fuzzwork data. Direct interaction with the Fuzzwork API (market.fuzzwork.co.uk/api/) may require specific query parameters or formats not detailed here. Related Datasets: - Market Order Snapshots (/datasets/market-orders) ``` -------------------------------- ### EVE Ref Killmail Data Source Source: https://docs.everef.net/datasets/killmails Details the primary source for EVE Ref killmail archives and its relationship with ESI and zKillboard. ```APIDOC Source: data.everef.net/killmails/ Description: Contains archives of all killmails from zKillboard. Each file in the archive contains all killmails for a specific day. Killmails are stored verbatim as they appear on the ESI's /killmails/{killmail_id}/{killmail_hash}/ endpoint. Files are updated in-place when new killmails are added for a day. Related Sources: zKillboard: https://zkillboard.com/ ESI (EVE Swagger Interface): https://esi.evetech.net/ui/ Data Fetching: EVE Ref fetches killmails directly from ESI using killmail IDs and hashes obtained from zKillboard. Extra information present on zKillboard's API is not included as EVE Ref does not scrape zKillboard's API directly for data content. ``` -------------------------------- ### EVE Ref Market History Scraping Source: https://docs.everef.net/commands/scrape-market-history Details the process and strategies for scraping EVE Online market history data. It explains the daily job, rate limiting challenges, batch processing, and intelligent data fetching using various sources to cover the vast number of region-type pairs. ```APIDOC ## Scrape Market History Produces the [market history dataset](/datasets/market-history). This job runs daily after downtime and takes almost all day to complete. This is due to the ESI market history endpoint being rate limited to 300 requests per minute (5 per second) and that each region-type pair has to be requested individually. There are roughly 1.7 million valid region-type pairs to try, with the crawler trying about 380k each day. It will prioritise more important region-type pairs to be fetched first, but it's essentially guessing what data to fetch. See [Sources](#sources) below for how it does that. In order for the crawler to be resilient to crashes and be able to resume where it left off, it processes these region-type pairs in batches. Each batch consists of 9,000 pairs. Once a batch has been scanned, any new daily files are uploaded to the archive. Each batch takes roughly 30 minutes to complete, meaning the first batch of data is available soon after downtime, and any consumers don't have to wait all day. ### Sources There are about 1.7 million valid region-type pairs to search though, which isn't feasible to do daily. Instead, the following "sources" are implemented to try and intelligently explore the space: * `HistoryRegionTypeSource`: Looks at the market history for the past 450 days. Any region-type pairs present in the history at any point will be tried again. These pairs are checked in market cap order, so highly traded items are checked first. * `ActiveOrdersRegionTypeSource`: Queries the [active types market endpoint](https://esi.evetech.net/ui/#/operations/Market/get_markets_region_id_types) for each region. * `HistoricalOrdersRegionTypeSource`: Looks at [market order snapshots](/datasets/market-orders) for the past 30 days. It downloads a random file for each day. This has a similar effect as active types, but looking into the past. * `TopTradedRegionTypeSource`: Also looks at market history for the past 450 days, but identifies the top market cap types and ensures those types are checked in every region. * `ExplorerRegionTypeSource`: Takes all 1.7 million pairs and groups them into 100 buckets, trying a different bucket each day. This guarantees that we're exploring the entire space at regular intervals. ### Additional sources on 2023-10-02 The run on 2023-10-02 saw the introduction of `HistoricalOrdersRegionTypeSource`, `TopTradedRegionTypeSource`, and `ExplorerRegionTypeSource`. This was in release [2.31.0](https://github.com/autonomouslogic/eve-ref/releases/tag/2.31.0). ``` -------------------------------- ### Market History Database Table Schema (SQL) Source: https://docs.everef.net/commands/import-market-history Defines the 'market_history' table structure, including columns for date, region, type, average price, highest price, lowest price, volume, order count, and HTTP last modified timestamp. It also includes primary key constraints and indexes for efficient querying. ```sql create table market_history ( date date not null, region_id integer not null, type_id integer not null, average numeric(20, 2) not null, highest numeric(20, 2) not null, lowest numeric(20, 2) not null, volume bigint not null, order_count integer not null, http_last_modified timestamp with time zone, primary key (date, region_id, type_id) ); create index market_history_region_id_type_id on market_history (region_id, type_id); create index market_history_type_id on market_history (type_id); ``` -------------------------------- ### Reference Data Download Source: https://docs.everef.net/datasets/reference-data The EVE Ref Reference Data is available for direct download. This dataset combines information from the EVE Online SDE, ESI, and Hoboleaks exports into a common format, serving as a unified source for game data. ```APIDOC Download Reference Data: URL: https://data.everef.net/reference-data/ Description: Full download of the EVE Ref Reference Data set. Content: Merged data from EVE Online SDE, ESI, and Hoboleaks exports. ``` -------------------------------- ### Import Market History Docker Command Source: https://docs.everef.net/commands/import-market-history This command imports market history data into a PostgreSQL database using Docker. It requires specific environment variables for database connection and configuration. The import process is designed to be incremental, only adding new data not already present in the database. The command is currently in development and requires careful database isolation. ```shell docker run -it --rm \ -e "DATABASE_URL=jdbc:postgresql://localhost:5432/everef" \ -e "DATABASE_USERNAME=everef" \ -e "DATABASE_PASSWORD=password1" \ -e "INSERT_SIZE=100000" \ autonomouslogic/eve-ref:latest \ import-market-history ``` -------------------------------- ### Configure EVE Ref Docker via .env File Source: https://docs.everef.net/commands/docker Sets environment variables for the Docker container configuration using the --env-file flag, referencing a local .env file. ```bash --env-file ``` -------------------------------- ### Sync Fuzzwork Ordersets Command Source: https://docs.everef.net/datasets/fuzzwork-ordersets Command to synchronize market order data from Fuzzwork. ```bash # Command to sync Fuzzwork Ordersets # This command is part of the EVE Ref project # It likely fetches data from Fuzzwork's market API # and stores it locally. # Example usage (hypothetical, as exact command is not provided): # eve-ref sync fuzzwork-ordersets ``` -------------------------------- ### EVE Ref Killmail Totals File Source: https://docs.everef.net/datasets/killmails Explains the purpose and usage of the totals.json file for tracking daily killmail counts. ```APIDOC File: totals.json (Located at: https://data.everef.net/killmails/totals.json) Description: Contains a count of all killmails stored per day. Since killmails are only ever added, this file can be used to determine which days have new killmails. Comparison: Akin to zKillboard's own totals.json file (https://zkillboard.com/api/history/totals.json). Usage: Monitor this file to identify days with updated or new killmail data. ``` -------------------------------- ### EVE Ref API Reference Source: https://docs.everef.net/api The EVE Ref API is accessible at api.everef.net. An OpenAPI specification is available for generating clients. The Industry Cost API is a specific endpoint for retrieving industry cost data. ```APIDOC API Base URL: api.everef.net OpenAPI Specification: Link: https://github.com/autonomouslogic/eve-ref/blob/main/spec/eve-ref-api.yaml Description: Provides the full API schema for client generation. Industry Cost API: Endpoint: /api/industry-cost Description: Provides data related to industry costs within EVE Online. Note: Further details on parameters and return values are available via the OpenAPI specification. ``` -------------------------------- ### Extract Unique Keys from JSON using jq Source: https://docs.everef.net/commands/build-ref-data This command uses `jq` to process a JSON file, extract all keys from each object within an array, and then sort them uniquely. It's useful for analyzing the structure of large JSON datasets. ```bash cat dogma-attributes.json | jq '.[] | keys | .[]' | sort -u ``` -------------------------------- ### EVE Ref Public Contracts contract_dynamic_items.csv Format Source: https://docs.everef.net/datasets/public-contracts Outlines the columns for contract_dynamic_items.csv, detailing dynamic item properties fetched via the Dogma API, with EVE Ref specific fields. ```CSV created_by,mutator_type_id,source_type_id,item_id,contract_id,http_last_modified ``` -------------------------------- ### Configure EVE Ref Docker via Environment Variable Source: https://docs.everef.net/commands/docker Sets an environment variable for the Docker container configuration using the -e flag. This is a common method for passing configuration values. ```bash -e "VARIABLE=value" ``` -------------------------------- ### EVE ESI Structure Endpoints Source: https://docs.everef.net/datasets/structures Details the EVE Swagger Interface (ESI) endpoints used by EVE Ref to scrape structure information. These endpoints provide data on universe structures and their associated markets. ```APIDOC Endpoint: /universe/structures/ Description: Retrieves a list of all structures in the universe. Parameters: None Returns: An array of structure objects. Endpoint: /universe/structures/{structure_id}/ Description: Retrieves details for a specific structure by its ID. Parameters: structure_id (integer): The ID of the structure to retrieve. Returns: A structure object with detailed information. Endpoint: /markets/structures/{structure_id}/ Description: Retrieves market information for a specific structure by its ID. Parameters: structure_id (integer): The ID of the structure to retrieve market data for. Returns: Market data for the specified structure. ``` -------------------------------- ### EVE Ref Invention Cost API Endpoint Source: https://docs.everef.net/api/industry-cost This section documents the primary API endpoint for calculating invention costs. It details the required and optional query parameters that influence the calculation, such as product ID, runs, system cost index, facility tax, and decryptor type. ```APIDOC API: /v1/industry/cost Method: GET Description: Calculates the cost of invention jobs based on product, runs, structure, system, and efficiency parameters. Query Parameters: - product_id (integer, required): The desired product type ID for the invention. - runs (integer, required): The number of invention runs to calculate. - decryptor_id (integer, required): The type ID of the decryptor to use for the invention. - blueprint_id (integer, optional): The blueprint ID to calculate. If not provided, it might be inferred or default. - me (integer, optional): The material efficiency (ME) level of the blueprint. Defaults to 0. - te (integer, optional): The time efficiency (TE) level of the blueprint. Defaults to 0. Structure Parameters: - structure_type_id (integer, optional): The type ID of the structure where the job is installed. If not set, an NPC station is assumed. - rig_id (integer, optional): The type IDs of the rigs installed on the structure. System Parameters: - system_id (integer, optional): The ID of the system where the job is installed. Used to resolve security class and cost indices. - security (float, optional): The security class of the system. Used if system_id is not provided. - manufacturing_cost (float, optional): The manufacturing cost index of the system. - invention_cost (float, optional): The invention cost index of the system. This is a critical parameter for invention cost calculation. - copying_cost (float, optional): The copying cost index of the system. - reaction_cost (float, optional): The reaction cost index of the system. - researching_me_cost (float, optional): The researching material efficiency cost index of the system. - researching_te_cost (float, optional): The researching time efficiency cost index of the system. Example Usage: https://api.everef.net/v1/industry/cost?product_id=2622&runs=10&structure_type_id=35825&decryptor_id=34206&invention_cost=0.1171&facility_tax=0.005 ``` -------------------------------- ### Market Order Snapshot CSV Format Source: https://docs.everef.net/datasets/market-orders Defines the columns included in the EVE Ref market order snapshot CSV files. It lists columns directly from the ESI endpoint and additional columns added by EVE Ref. ```APIDOC MarketOrderSnapshotColumns: ESI Columns: duration: Order duration in days is_buy_order: Boolean indicating if it's a buy order issued: Timestamp of when the order was issued location_id: ID of the location (station or structure) min_volume: Minimum volume required for the order order_id: Unique identifier for the order price: The price of the order range: The range of the order (e.g., 'station', 'sol', '1' to '32767') system_id: The system ID where the order is located type_id: The item type ID volume_remain: The remaining volume of the order volume_total: The total volume of the order EVE Ref Added Columns: http_last_modified: HTTP Last-Modified header value from ESI region_id: The region ID where the order is located station_id: The station ID where the order is located (if applicable) constellation_id: The constellation ID where the order is located ```