EVE Ref Docs

https://docs.everef.net
Admin
EVE Ref is a reference site for EVE Online, providing access to industry cost calculations, merged...
Tokens:20,527
Snippets:51
Trust Score:8.1
Update:1 year ago
Show doc for...
Tokens:
Raw
### Docker Command Example

Source: https://docs.everef.net/commands/scrape-market-history

An example command for interacting with Docker, as listed under the project's commands.

```bash
docker run --rm -v $(pwd):/app your-eve-ref-image docker-command-here
```

--------------------------------

### Run EVE Ref API Locally

Source: https://docs.everef.net/api

This command allows you to run the EVE Ref API locally using Docker. It pulls the latest image and starts the API service.

```docker
docker run -it --rm autonomouslogic/eve-ref:latest api

```

--------------------------------

### Sync Market Orders with rclone on Docker

Source: https://docs.everef.net/datasets/downloading-datasets

This command demonstrates how to sync market order history data using rclone within a Docker container. It mounts the current directory to persist downloaded data. Requires Docker to be installed.

```shell
docker run --rm -v $(pwd):/data -w /data rclone/rclone sync -v --checkers 2 --transfers 2 --multi-thread-streams 0 --http-url https://data.everef.net :http:/market-orders/history/2023 pathtosyncto
```

--------------------------------

### Sync Market Orders with rclone

Source: https://docs.everef.net/datasets/downloading-datasets

This command uses rclone to synchronize market order history data for a specific year. It leverages multiple concurrent downloads for efficiency. Ensure rclone is installed and configured.

```shell
rclone sync -v --checkers 2 --transfers 2 --multi-thread-streams 0 --http-url https://data.everef.net :http:/market-orders/history/2023 pathtosync
```

--------------------------------

### Mount Data Directory with rclone

Source: https://docs.everef.net/datasets/downloading-datasets

This command mounts the entire data directory from data.everef.net onto the local filesystem using rclone. It configures caching and timeouts for optimal performance. Useful for accessing data as if it were local.

```shell
rclone mount --attr-timeout 1m --dir-cache-time 1m --vfs-cache-mode full --checkers 2 --transfers 2 --multi-thread-streams 0 --http-url https://data.everef.net :http:/ pathtomount
```

--------------------------------

### Download Market Orders History using Docker

Source: https://docs.everef.net/datasets/downloading-datasets

This snippet demonstrates how to use the `mwendler/wget` Docker image to download market order history data from data.everef.net. It mounts the current directory to `/data` inside the container, sets the working directory, and uses `wget` with specific flags to recursively download files, excluding `index.html`, and disabling certificate checks.

```shell
docker run --rm -v $(pwd):/data -w /data mwendler/wget -r -np -N -nv --domains=data.everef.net -R index.html --no-check-certificate https://data.everef.net/market-orders/history/2023/
```

--------------------------------

### Fetch Market Structures

Source: https://docs.everef.net/datasets/structures

This command fetches the latest market structure data from Everef.net and filters for structures that are marked as market structures using jq. It requires curl and jq to be installed.

```bash
curl -s https://data.everef.net/structures/structures-latest.v2.json | jq '.[] | select(.is_market_structure)'
```

--------------------------------

### Download Market Orders with wget

Source: https://docs.everef.net/datasets/downloading-datasets

This command uses wget to recursively download market order data for a specific year. It includes options to prevent ascending directories, enforce timestamp checking, and restrict downloads to the specified domain. Note that wget does not support multiple concurrent downloads.

```shell
wget -r -np -N -nv --domains=data.everef.net -R index.html https://data.everef.net/market-orders/history/2023/
```

--------------------------------

### Configure Initial Market History Import Date

Source: https://docs.everef.net/commands/import-market-history

Set the environment variable IMPORT_MARKET_HISTORY_MIN_DATE to '2003-01-01' to import all market history data starting from January 1, 2003. This variable can be omitted to use the default behavior.

```shell
IMPORT_MARKET_HISTORY_MIN_DATE=2003-01-01
```

--------------------------------

### Invention Cost Calculation Response Example

Source: https://docs.everef.net/api/industry-cost

This JSON object represents a typical response from the invention cost API endpoint. It provides detailed breakdown of costs, materials, probabilities, and time estimations for an invention job.

```JSON
{
  "invention" : {
    "2622": {
      "product_id": 2622,
      "runs": 10,
      "time": "PT136H40M23S",
      "materials": {
        "20418": {
          "type_id": 20418,
          "quantity": 10,
          "cost": 1043000.50
        },
        "20420": {
          "type_id": 20420,
          "quantity": 10,
          "cost": 1020634.60
        },
        "34206": {
          "type_id": 34206,
          "quantity": 10,
          "cost": 4285722.00
        }
      },
      "estimated_item_value": 16852252,
      "system_cost_index": 39468,
      "system_cost_bonuses": -1184,
      "facility_tax": 1685,
      "scc_surcharge": 13482,
      "alpha_clone_tax": 0,
      "total_job_cost": 53451,
      "total_material_cost": 6349357.10,
      "total_cost": 6402808.10,
      "blueprint_id": 805,
      "probability": 0.49583333333333335,
      "runs_per_copy": 12,
      "units_per_run": 5000,
      "expected_copies": 4.958333333333334,
      "expected_runs": 59.50000000000001,
      "expected_units": 297500.00000000006,
      "me": 3,
      "te": 12,
      "job_cost_base": 337045,
      "avg_time_per_copy": "PT27H33M51.529S",
      "avg_time_per_run": "PT2H17M49.294S",
      "avg_time_per_unit": "PT1.653S",
      "avg_cost_per_copy": 1291322.64,
      "avg_cost_per_run": 107610.22,
      "avg_cost_per_unit": 21.52
    }
  }
}
```

--------------------------------

### EVE Ref REST API Endpoints

Source: https://docs.everef.net/datasets/reference-data

Provides access to various EVE Online game data resources. The full OpenAPI specification is available on GitHub. Examples include fetching lists of resources or specific items by ID.

```APIDOC
OpenAPI Specification:
  URL: https://github.com/autonomouslogic/eve-ref/blob/main/spec/reference-data.yaml

Resource Paths:
  - /blueprints
    Description: Retrieves a list of all blueprints.
    Example: https://ref-data.everef.net/blueprints
  - /blueprints/{id}
    Description: Retrieves a specific blueprint by its ID.
    Example: https://ref-data.everef.net/blueprints/999
  - /categories
    Description: Retrieves a list of all item categories.
    Example: https://ref-data.everef.net/categories
  - /categories/{id}
    Description: Retrieves a specific item category by its ID.
    Example: https://ref-data.everef.net/categories/4
  - /dogma_attributes
    Description: Retrieves a list of all Dogma attributes.
    Example: https://ref-data.everef.net/dogma_attributes
  - /dogma_attributes/{id}
    Description: Retrieves a specific Dogma attribute by its ID.
    Example: https://ref-data.everef.net/dogma_attributes/37
  - /dogma_effects
    Description: Retrieves a list of all Dogma effects.
    Example: https://ref-data.everef.net/dogma_effects
  - /dogma_effects/{id}
    Description: Retrieves a specific Dogma effect by its ID.
    Example: https://ref-data.everef.net/dogma_effects/11
  - /groups
    Description: Retrieves a list of all item groups.
    Example: https://ref-data.everef.net/groups
  - /groups/{id}
    Description: Retrieves a specific item group by its ID.
    Example: https://ref-data.everef.net/groups/18
  - /market_groups
    Description: Retrieves a list of all market groups.
    Example: https://ref-data.everef.net/market_groups
  - /market_groups/{id}
    Description: Retrieves a specific market group by its ID.
    Example: https://ref-data.everef.net/market_groups/1857
  - /meta_groups
    Description: Retrieves a list of all meta groups.
    Example: https://ref-data.everef.net/meta_groups
  - /meta_group/{id}
    Description: Retrieves a specific meta group by its ID.
    Example: https://ref-data.everef.net/meta_group/6
  - /mutaplasmids
    Description: Retrieves a list of all mutaplasmids.
    Example: https://ref-data.everef.net/mutaplasmids
  - /mutaplasmids/{id}
    Description: Retrieves a specific mutaplasmid by its ID.
    Example: https://ref-data.everef.net/mutaplasmids/52225
  - /regions
    Description: Retrieves a list of all regions.
    Example: https://ref-data.everef.net/regions
  - /regions/{id}
    Description: Retrieves a specific region by its ID.
    Example: https://ref-data.everef.net/regions/10000002
  - /schematics
    Description: Retrieves a list of all schematics.
    Example: https://ref-data.everef.net/schematics
  - /schematics/{id}
    Description: Retrieves a specific schematic by its ID.
    Example: https://ref-data.everef.net/schematics/65
  - /skills
    Description: Retrieves a list of all skills.
    Example: https://ref-data.everef.net/skills
  - /skills/{id}
    Description: Retrieves a specific skill by its ID.
    Example: https://ref-data.everef.net/skills/3336
  - /types
    Description: Retrieves a list of all item types.
    Example: https://ref-data.everef.net/types
  - /types/{id}
    Description: Retrieves a specific item type by its ID.
    Example: https://ref-data.everef.net/types/645
  - /units
    Description: Retrieves a list of all units.
    Example: https://ref-data.everef.net/units
  - /units/{id}
    Description: Retrieves a specific unit by its ID.
    Example: https://ref-data.everef.net/units/1
```

--------------------------------

### BuildRefData CLI Command and Data Loading Process

Source: https://docs.everef.net/commands/build-ref-data

Details the core functionality of the `BuildRefData` CLI command. It outlines the configuration via `refdata.yaml`, the roles of `SdeLoader` and `EsiLoader` in reading data, the use of MVStore for intermediate storage, and the final merging process to create the reference data file.

```APIDOC
BuildRefData:
  Description: Builds and uploads the main reference data file.
  Configuration: Uses `refdata.yaml` for processing sources.
  Data Loading:
    - `SdeLoader`: Loads data from the SDE (Static Data Export).
    - `EsiLoader`: Loads data from the ESI (EVE Swagger Interface).
    - Loaders read input files and output objects.
  Storage:
    - MVStore is used to store intermediate objects.
    - Maps are created for each type and source (e.g., `types/SDE`, `types/ESI`).
  Merging:
    - All loader outputs are merged into the final reference data.
    - Loaders output objects in their final, ready-to-merge form.
```

--------------------------------

### EVE Ref Commands

Source: https://docs.everef.net/commands/scrape-market-history

Lists various command-line utilities provided by the EVE Ref project for data management and processing. Includes commands for Docker operations, data building, and scraping market data.

```APIDOC
## Commands

*   `Docker`: Commands related to Docker operations.
*   `Field Replication`: Commands for handling field replication.
*   `Build Ref Data`: Commands to build reference data.
*   `Import Market History`: Commands to import market history data.
*   `Scrape Market History`: Commands to scrape market history data.
    *   `Sources`: Details on data sources used for scraping.
    *   `Additional sources on 2023-10-02`: Information on new sources added on a specific date.
*   `Scrape Market Orders`: Commands to scrape market order data.
*   `Scrape Public Contracts`: Commands to scrape public contract data.
*   `Sync Fuzzwork Ordersets`: Commands to synchronize Fuzzwork ordersets.
```

--------------------------------

### EVE Ref Industry Cost API Endpoint

Source: https://docs.everef.net/api/industry-cost

Provides accurate industry calculations for EVE Online manufacturing and invention. Accepts GET requests and requires specific item or blueprint IDs. Designed for ease of use from any programming language.

```APIDOC
Endpoint: https://api.everef.net/v1/industry/cost
Method: GET

Description:
Calculates industry costs for EVE Online items, supporting manufacturing and invention scenarios.

Parameters:
- product_id (integer, required): The item ID for which to calculate manufacturing costs. If used for invention, this is the blueprint to be invented.
- blueprint_id (integer, optional): The source blueprint type ID. Used to calculate all activities on a specific blueprint, or when an item can come from multiple blueprints.

Functionality:
- For manufacture: Supply the `product_id` parameter of an item.
- For manufacture of items requiring invention: Supply the `product_id` parameter of the item; invention will be automatically calculated.
- For invention only: Supply the `product_id` parameter of the blueprint to be invented.
- For items from multiple blueprints: Set `blueprint_id` as the source blueprint type ID.
- To calculate all activities on a specific blueprint: Set `blueprint_id`.

Returns:
JSON object containing detailed cost calculations, material requirements, and invention outcomes.
```

--------------------------------

### Running the EVE Ref API Locally

Source: https://docs.everef.net/api/industry-cost

This command initiates the EVE Ref API container on your local machine using Docker. It pulls the latest 'latest' tag of the 'autonomouslogic/eve-ref' image, runs it interactively, and automatically removes the container once it stops.

```docker
docker run -it --rm autonomouslogic/eve-ref:latest api
```

--------------------------------

### Build EVE Ref Docker Image

Source: https://docs.everef.net/commands/docker

Command to build the Docker image for EVE Ref using the provided Makefile.

```makefile
make docker
```

--------------------------------

### Get Public Contract Bids by Contract ID

Source: https://docs.everef.net/datasets/public-contracts

Retrieves all bid entries for a given public contract ID from EVE Online. This API includes standard bid details and additional fields provided by EVE Ref for enhanced data management.

```APIDOC
GET /contracts/public/bids/{contract_id}/
  Description: Retrieves all bid entries for a given public contract ID.
  Parameters:
    - contract_id (integer): The ID of the contract.
  ESI Fields:
    - amount (number): The bid amount.
    - bid_id (integer): The ID of the bid.
    - date_bid (string): The date the bid was placed (ISO 8601 format).
  EVE Ref Added Fields:
    - http_last_modified (string): Timestamp of the last HTTP modification.
    - contract_id (integer): Identifier for the contract.
```

--------------------------------

### Get Dogma Dynamic Item Effects by Type and Item

Source: https://docs.everef.net/datasets/public-contracts

Retrieves dynamic item effects for a specific item type and ID from the EVE Online Dogma system. This endpoint provides base ESI fields and additional columns added by EVE Ref for contract tracking.

```APIDOC
GET /dogma/dynamic/items/{type_id}/{item_id}/
  Description: Retrieves dynamic item effects for a specific item type and ID.
  Parameters:
    - type_id (integer): The ID of the item type.
    - item_id (integer): The ID of the specific item.
  ESI Fields:
    - effect_id (integer): The ID of the dogma effect.
    - is_default (boolean): Indicates if the effect is default.
  EVE Ref Added Fields:
    - contract_id (integer): Identifier for the contract.
    - item_id (integer): Identifier for the item.
    - http_last_modified (string): Timestamp of the last HTTP modification.
```

--------------------------------

### Adding New Resources to BuildRefData

Source: https://docs.everef.net/commands/build-ref-data

Provides instructions on how to add new data resources to the EVE Ref project. This involves updating loaders, adding sample data, and ensuring model consistency.

```APIDOC
Adding New Resources:
  Steps:
    1. Set up resources in ESI and SDE loaders.
    2. Add samples to relevant resource directories.
    3. Add tests for both loaders and the main command.
  Sample Sources:
    - Input samples: Directly from ESI scrape or SDE dump.
    - Output samples: Constructed manually.
  Verification:
    - Add to `PublishRefData` and `PublishRefDataTest`.
    - Run `VerifyRefDataModels` to ensure model existence.
```

--------------------------------

### Testing BuildRefData

Source: https://docs.everef.net/commands/build-ref-data

Describes the testing strategy for the `BuildRefData` command, including the structure of test resources and the classes used for validation. It covers end-to-end testing and specific loader tests.

```APIDOC
BuildRefData Testing:
  Test Resources:
    - `resources/refdata/esi`: Samples of ESI files.
    - `resources/refdata/sde`: Samples of SDE files.
    - `resources/refdata/refdata`: Expected final format (loaded, transformed, merged).
  Testing Classes:
    - `BuildRefDataTest`: Performs full end-to-end testing.
    - `EsiLoaderTest`: Tests the `EsiLoader` with contextual resources.
    - `SdeLoaderTest`: Tests the `SdeLoader` with contextual resources.
  Intermediate Output Testing:
    - Samples for intermediate loader outputs are available in contextual resource folders.
```

--------------------------------

### Everef API: Calculate Manufacturing Cost

Source: https://docs.everef.net/api/industry-cost

This API endpoint calculates the estimated cost for manufacturing items in the game EVE Online. It requires parameters specifying the product, manufacturing runs, efficiency levels (ME/TE), structure type, security status, installed rigs, and various cost modifiers. The response provides a detailed breakdown of material costs, job costs, and estimated item value, including potential invention steps.

```APIDOC
GET https://api.everef.net/v1/industry/cost?product_id=22430&runs=8&me=4&te=4&structure_type_id=35827&security=LOW_SEC&rig_id=37180&rig_id=37183&system_cost_bonus=-0.5&manufacturing_cost=0.0129&facility_tax=0.02

Parameters:
- product_id: The ID of the item to manufacture.
- runs: The number of manufacturing runs.
- me: The Material Efficiency level.
- te: The Time Efficiency level.
- structure_type_id: The ID of the structure used for manufacturing.
- security: The security status of the system (e.g., LOW_SEC).
- rig_id: IDs of installed rigs (can be specified multiple times).
- system_cost_bonus: The system cost bonus applied.
- manufacturing_cost: The system's manufacturing cost index.
- facility_tax: The tax applied by the facility.

Response Structure (JSON):
{
  "manufacturing" : {
    "[product_id]" : {
      "product_id" : integer,
      "runs" : number,
      "time" : string, // ISO 8601 duration format
      "materials" : {
        "[type_id]" : {
          "type_id" : integer,
          "quantity" : number,
          "cost_per_unit" : number,
          "cost" : number
        }
      },
      "estimated_item_value" : number,
      "system_cost_index" : number,
      "system_cost_bonuses" : number,
      "facility_tax" : number,
      "scc_surcharge" : number,
      "alpha_clone_tax" : number,
      "total_job_cost" : number,
      "total_material_cost" : number,
      "total_cost" : number,
      "blueprint_id" : integer,
      "units" : number,
      "units_per_run" : number,
      "time_per_run" : string, // ISO 8601 duration format
      "time_per_unit" : string, // ISO 8601 duration format
      "total_cost_per_run" : number,
      "total_cost_per_unit" : number
    }
  },
  "invention" : {
    "[blueprint_id]" : {
      "product_id" : integer,
      "runs" : number,
      "time" : string, // ISO 8601 duration format
      "materials" : {
        "[type_id]" : {
          "type_id" : integer,
          "quantity" : number,
          "cost_per_unit" : number,
          "cost" : number
        }
      },
      "estimated_item_value" : number,
      "system_cost_index" : number,
      "system_cost_bonuses" : number,
      "facility_tax" : number,
      "scc_surcharge" : number,
      "alpha_clone_tax" : number,
      "total_job_cost" : number,
      "total_material_cost" : number,
      "total_cost" : number,
      "blueprint_id" : integer,
      "probability" : number,
      "runs_per_copy" : number,
      "units_per_run" : number,
      "expected_copies" : number,
      "expected_runs" : number,
      "expected_units" : number,
      "me" : integer,
      "te" : integer,
      "job_cost_base" : number,
      "avg_time_per_copy" : string, // ISO 8601 duration format
      "avg_time_per_run" : string, // ISO 8601 duration format
      "avg_time_per_unit" : string, // ISO 8601 duration format
      "avg_cost_per_copy" : number,
      "avg_cost_per_run" : number,
      "avg_cost_per_unit" : number
    }
  }
}
```

--------------------------------

### Run EVE Ref Docker Container

Source: https://docs.everef.net/commands/docker

Command to execute the EVE Ref Docker image. Replace 'command' with the desired operation to perform inside the container.

```bash
docker run autonomouslogic/everef:latest command
```

--------------------------------

### Fuzzwork Market Orders API Reference

Source: https://docs.everef.net/datasets/fuzzwork-ordersets

Reference to the Fuzzwork Market Orders API, which provides EVE Online market data. The EVE Ref project mirrors this data.

```APIDOC
API Endpoint:
https://data.everef.net/fuzzwork/ordersets/

Description:
This endpoint mirrors the market orders data provided by Fuzzwork.
It is similar to the Market Order Snapshots dataset.

Source API:
https://market.fuzzwork.co.uk/api/

Data Provided:
- Market order details for EVE Online.
- Likely includes information such as item type, price, quantity, location, etc.

Usage:
Accessing the data.everef.net URL provides a mirror of the Fuzzwork data.
Direct interaction with the Fuzzwork API (market.fuzzwork.co.uk/api/) may require specific query parameters or formats not detailed here.

Related Datasets:
- Market Order Snapshots (/datasets/market-orders)
```

--------------------------------

### EVE Ref Killmail Data Source

Source: https://docs.everef.net/datasets/killmails

Details the primary source for EVE Ref killmail archives and its relationship with ESI and zKillboard.

```APIDOC
Source:
  data.everef.net/killmails/

Description:
  Contains archives of all killmails from zKillboard.
  Each file in the archive contains all killmails for a specific day.
  Killmails are stored verbatim as they appear on the ESI's /killmails/{killmail_id}/{killmail_hash}/ endpoint.
  Files are updated in-place when new killmails are added for a day.

Related Sources:
  zKillboard: https://zkillboard.com/
  ESI (EVE Swagger Interface): https://esi.evetech.net/ui/

Data Fetching:
  EVE Ref fetches killmails directly from ESI using killmail IDs and hashes obtained from zKillboard.
  Extra information present on zKillboard's API is not included as EVE Ref does not scrape zKillboard's API directly for data content.
```

--------------------------------

### EVE Ref Market History Scraping

Source: https://docs.everef.net/commands/scrape-market-history

Details the process and strategies for scraping EVE Online market history data. It explains the daily job, rate limiting challenges, batch processing, and intelligent data fetching using various sources to cover the vast number of region-type pairs.

```APIDOC
## Scrape Market History

Produces the [market history dataset](/datasets/market-history).

This job runs daily after downtime and takes almost all day to complete.
This is due to the ESI market history endpoint being rate limited to 300 requests per minute (5 per second)
and that each region-type pair has to be requested individually.
There are roughly 1.7 million valid region-type pairs to try, with the crawler trying about 380k each day.
It will prioritise more important region-type pairs to be fetched first, but it's essentially guessing what data to fetch.
See [Sources](#sources) below for how it does that.

In order for the crawler to be resilient to crashes and be able to resume where it left off,
it processes these region-type pairs in batches.
Each batch consists of 9,000 pairs.
Once a batch has been scanned, any new daily files are uploaded to the archive.
Each batch takes roughly 30 minutes to complete, meaning the first batch of data is available soon after downtime,
and any consumers don't have to wait all day.

### Sources

There are about 1.7 million valid region-type pairs to search though, which isn't feasible to do daily.
Instead, the following "sources" are implemented to try and intelligently explore the space:

*   `HistoryRegionTypeSource`: Looks at the market history for the past 450 days. Any region-type pairs present in the history at any point will be tried again. These pairs are checked in market cap order, so highly traded items are checked first.
*   `ActiveOrdersRegionTypeSource`: Queries the [active types market endpoint](https://esi.evetech.net/ui/#/operations/Market/get_markets_region_id_types) for each region.
*   `HistoricalOrdersRegionTypeSource`: Looks at [market order snapshots](/datasets/market-orders) for the past 30 days. It downloads a random file for each day. This has a similar effect as active types, but looking into the past.
*   `TopTradedRegionTypeSource`: Also looks at market history for the past 450 days, but identifies the top market cap types and ensures those types are checked in every region.
*   `ExplorerRegionTypeSource`: Takes all 1.7 million pairs and groups them into 100 buckets, trying a different bucket each day. This guarantees that we're exploring the entire space at regular intervals.

### Additional sources on 2023-10-02

The run on 2023-10-02 saw the introduction of `HistoricalOrdersRegionTypeSource`, `TopTradedRegionTypeSource`, and `ExplorerRegionTypeSource`.
This was in release [2.31.0](https://github.com/autonomouslogic/eve-ref/releases/tag/2.31.0).
```

--------------------------------

### Market History Database Table Schema (SQL)

Source: https://docs.everef.net/commands/import-market-history

Defines the 'market_history' table structure, including columns for date, region, type, average price, highest price, lowest price, volume, order count, and HTTP last modified timestamp. It also includes primary key constraints and indexes for efficient querying.

```sql
create table market_history
(
    date               date           not null,
    region_id          integer        not null,
    type_id            integer        not null,
    average            numeric(20, 2) not null,
    highest            numeric(20, 2) not null,
    lowest             numeric(20, 2) not null,
    volume             bigint         not null,
    order_count        integer        not null,
    http_last_modified timestamp with time zone,
    primary key (date, region_id, type_id)
);

create index market_history_region_id_type_id
    on market_history (region_id, type_id);

create index market_history_type_id
    on market_history (type_id);
```

--------------------------------

### Reference Data Download

Source: https://docs.everef.net/datasets/reference-data

The EVE Ref Reference Data is available for direct download. This dataset combines information from the EVE Online SDE, ESI, and Hoboleaks exports into a common format, serving as a unified source for game data.

```APIDOC
Download Reference Data:
  URL: https://data.everef.net/reference-data/
  Description: Full download of the EVE Ref Reference Data set.
  Content: Merged data from EVE Online SDE, ESI, and Hoboleaks exports.
```

--------------------------------

### Import Market History Docker Command

Source: https://docs.everef.net/commands/import-market-history

This command imports market history data into a PostgreSQL database using Docker. It requires specific environment variables for database connection and configuration. The import process is designed to be incremental, only adding new data not already present in the database. The command is currently in development and requires careful database isolation.

```shell
docker run -it --rm \
	-e "DATABASE_URL=jdbc:postgresql://localhost:5432/everef" \
	-e "DATABASE_USERNAME=everef" \
	-e "DATABASE_PASSWORD=password1" \
	-e "INSERT_SIZE=100000" \
	autonomouslogic/eve-ref:latest \
	import-market-history

```

--------------------------------

### Configure EVE Ref Docker via .env File

Source: https://docs.everef.net/commands/docker

Sets environment variables for the Docker container configuration using the --env-file flag, referencing a local .env file.

```bash
--env-file
```

--------------------------------

### Sync Fuzzwork Ordersets Command

Source: https://docs.everef.net/datasets/fuzzwork-ordersets

Command to synchronize market order data from Fuzzwork.

```bash
# Command to sync Fuzzwork Ordersets
# This command is part of the EVE Ref project
# It likely fetches data from Fuzzwork's market API
# and stores it locally.

# Example usage (hypothetical, as exact command is not provided):
# eve-ref sync fuzzwork-ordersets
```

--------------------------------

### EVE Ref Killmail Totals File

Source: https://docs.everef.net/datasets/killmails

Explains the purpose and usage of the totals.json file for tracking daily killmail counts.

```APIDOC
File:
  totals.json
  (Located at: https://data.everef.net/killmails/totals.json)

Description:
  Contains a count of all killmails stored per day.
  Since killmails are only ever added, this file can be used to determine which days have new killmails.

Comparison:
  Akin to zKillboard's own totals.json file (https://zkillboard.com/api/history/totals.json).

Usage:
  Monitor this file to identify days with updated or new killmail data.
```

--------------------------------

### EVE Ref API Reference

Source: https://docs.everef.net/api

The EVE Ref API is accessible at api.everef.net. An OpenAPI specification is available for generating clients. The Industry Cost API is a specific endpoint for retrieving industry cost data.

```APIDOC
API Base URL:
  api.everef.net

OpenAPI Specification:
  Link: https://github.com/autonomouslogic/eve-ref/blob/main/spec/eve-ref-api.yaml
  Description: Provides the full API schema for client generation.

Industry Cost API:
  Endpoint: /api/industry-cost
  Description: Provides data related to industry costs within EVE Online.
  Note: Further details on parameters and return values are available via the OpenAPI specification.
```

--------------------------------

### Extract Unique Keys from JSON using jq

Source: https://docs.everef.net/commands/build-ref-data

This command uses `jq` to process a JSON file, extract all keys from each object within an array, and then sort them uniquely. It's useful for analyzing the structure of large JSON datasets.

```bash
cat dogma-attributes.json | jq '.[] | keys | .[]' | sort -u
```

--------------------------------

### EVE Ref Public Contracts contract_dynamic_items.csv Format

Source: https://docs.everef.net/datasets/public-contracts

Outlines the columns for contract_dynamic_items.csv, detailing dynamic item properties fetched via the Dogma API, with EVE Ref specific fields.

```CSV
created_by,mutator_type_id,source_type_id,item_id,contract_id,http_last_modified
```

--------------------------------

### Configure EVE Ref Docker via Environment Variable

Source: https://docs.everef.net/commands/docker

Sets an environment variable for the Docker container configuration using the -e flag. This is a common method for passing configuration values.

```bash
-e "VARIABLE=value"
```

--------------------------------

### EVE ESI Structure Endpoints

Source: https://docs.everef.net/datasets/structures

Details the EVE Swagger Interface (ESI) endpoints used by EVE Ref to scrape structure information. These endpoints provide data on universe structures and their associated markets.

```APIDOC
Endpoint: /universe/structures/
  Description: Retrieves a list of all structures in the universe.
  Parameters:
    None
  Returns:
    An array of structure objects.

Endpoint: /universe/structures/{structure_id}/
  Description: Retrieves details for a specific structure by its ID.
  Parameters:
    structure_id (integer): The ID of the structure to retrieve.
  Returns:
    A structure object with detailed information.

Endpoint: /markets/structures/{structure_id}/
  Description: Retrieves market information for a specific structure by its ID.
  Parameters:
    structure_id (integer): The ID of the structure to retrieve market data for.
  Returns:
    Market data for the specified structure.
```

--------------------------------

### EVE Ref Invention Cost API Endpoint

Source: https://docs.everef.net/api/industry-cost

This section documents the primary API endpoint for calculating invention costs. It details the required and optional query parameters that influence the calculation, such as product ID, runs, system cost index, facility tax, and decryptor type.

```APIDOC
API: /v1/industry/cost

Method: GET

Description: Calculates the cost of invention jobs based on product, runs, structure, system, and efficiency parameters.

Query Parameters:
  - product_id (integer, required): The desired product type ID for the invention.
  - runs (integer, required): The number of invention runs to calculate.
  - decryptor_id (integer, required): The type ID of the decryptor to use for the invention.
  - blueprint_id (integer, optional): The blueprint ID to calculate. If not provided, it might be inferred or default.
  - me (integer, optional): The material efficiency (ME) level of the blueprint. Defaults to 0.
  - te (integer, optional): The time efficiency (TE) level of the blueprint. Defaults to 0.

Structure Parameters:
  - structure_type_id (integer, optional): The type ID of the structure where the job is installed. If not set, an NPC station is assumed.
  - rig_id (integer, optional): The type IDs of the rigs installed on the structure.

System Parameters:
  - system_id (integer, optional): The ID of the system where the job is installed. Used to resolve security class and cost indices.
  - security (float, optional): The security class of the system. Used if system_id is not provided.
  - manufacturing_cost (float, optional): The manufacturing cost index of the system.
  - invention_cost (float, optional): The invention cost index of the system. This is a critical parameter for invention cost calculation.
  - copying_cost (float, optional): The copying cost index of the system.
  - reaction_cost (float, optional): The reaction cost index of the system.
  - researching_me_cost (float, optional): The researching material efficiency cost index of the system.
  - researching_te_cost (float, optional): The researching time efficiency cost index of the system.

Example Usage:
https://api.everef.net/v1/industry/cost?product_id=2622&runs=10&structure_type_id=35825&decryptor_id=34206&invention_cost=0.1171&facility_tax=0.005
```

--------------------------------

### Market Order Snapshot CSV Format

Source: https://docs.everef.net/datasets/market-orders

Defines the columns included in the EVE Ref market order snapshot CSV files. It lists columns directly from the ESI endpoint and additional columns added by EVE Ref.

```APIDOC
MarketOrderSnapshotColumns:
  ESI Columns:
    duration: Order duration in days
    is_buy_order: Boolean indicating if it's a buy order
    issued: Timestamp of when the order was issued
    location_id: ID of the location (station or structure)
    min_volume: Minimum volume required for the order
    order_id: Unique identifier for the order
    price: The price of the order
    range: The range of the order (e.g., 'station', 'sol', '1' to '32767')
    system_id: The system ID where the order is located
    type_id: The item type ID
    volume_remain: The remaining volume of the order
    volume_total: The total volume of the order
  EVE Ref Added Columns:
    http_last_modified: HTTP Last-Modified header value from ESI
    region_id: The region ID where the order is located
    station_id: The station ID where the order is located (if applicable)
    constellation_id: The constellation ID where the order is located
```
Tokens:
Raw