### Install Git Hooks

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install pre-commit Git hooks for local development.

```sh
PYTHONNOUSERSITE=1 pre-commit install --install-hooks
```

--------------------------------

### Install All Optional Dependencies

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/REFERENCE_INDEX.md

Install all available optional dependencies for histdatacom.

```sh
pip install histdatacom[all]
```

--------------------------------

### Using 'start' keyword for data range

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Use the 'start' keyword with the --start_yearmonth flag and specify --end_yearmonth to define a data range.

```txt
histdatacom -p audusd -f metatrader -s start -e 2008-12
```

--------------------------------

### Install from TestPyPI

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

After a TestPyPI dry run, use this command to install the package from TestPyPI in a disposable environment. This verifies the package installation and its dependencies resolve correctly from the test repository.

```sh
bash pypi.sh testpypi_install
```

--------------------------------

### Basic CLI Usage Example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Illustrates a typical command structure for fetching and processing data. This example shows how to specify pairs, formats, and timeframes.

```txt
usage: histdatacom [-h] [-A] [-U] [--by BY] [--version] [-V] [-D] [-X] [-C] [--config PATH] [-p PAIR [PAIR ...]] [--pair-groups GROUP [GROUP ...]] [-f FORMAT [FORMAT ...]] [-t TIMEFRAME [TIMEFRAME ...]] [-s START_YEARMONTH] [-e END_YEARMONTH] [-I] [-d] [-b BATCH_SIZE] [-c CPU_UTILIZATION]
                   [--data-directory DATA_DIRECTORY] [-v] [--orchestration-start] [--no-orchestration-start] [--submit-only] [--quality] [--repo-quality] [--repo-quality-columns] [--quality-target PATH [PATH ...]] [--quality-checks GROUP [GROUP ...]]
                   [--quality-report PATH] [--quality-profile PATH] [--quality-fail-on SEVERITY] [--quality-max-errors COUNT] [--quality-max-warnings COUNT]

options:
  -h, --help            show this help message and exit

Mode:
  -V, --validate_urls   Check generated list of URLs as valid download
                        locations
  -D, --download_data_archives
                        download specified pairs/formats/timeframe and create
                        data files
  -X, --extract_csvs    histdata.com delivers zip files. Use the -X flag to
                        extract them.
  -C, --build-cache, --cache-only, --build_cache
                        build canonical Polars .data caches and remove
                        transient ZIP/CSV sources after each cache is ready

Config:
  --config PATH         read recurrent-run defaults from a YAML file; explicit
                        CLI flags override configured values
  -p, --pairs PAIR [PAIR ...]
                        space separated currency pairs. e.g. -p eurusd usdjpy
                        ...
  --pair-groups, --instrument-groups, --symbol-groups GROUP [GROUP ...]
                        named instrument groups to union with --pairs.
                        Common groups: majors, minors, crosses, exotics,
                        metals, commodities, indices
  -f, --formats FORMAT [FORMAT ...]
                        space separated formats. -f metatrader ascii
                        ninjatrader metastock
  -t, --timeframes TIMEFRAME [TIMEFRAME ...]
                        space separated Timeframes. -t tick-data-quotes
                        1-minute-bar-quotes
  -s, --start_yearmonth START_YEARMONTH
                        set a start year and month for data. e.g. -s 2000-04
                        or -s 2015-00
  -e, --end_yearmonth END_YEARMONTH
                        set an end year and month for data. e.g. -e 2020-00
                        or -e 2022-04

Influxdb:
  -I, --import_to_influxdb
                        import data to influxdb instance. Use influxdb.yaml to
                        configure.
  -d, --delete_after_influx
                        delete data files after upload to influxdb
  -b, --batch_size BATCH_SIZE
                        (integer) influxdb write_api batch size. defaults to
                        5000

System:
  -c, --cpu_utilization CPU_UTILIZATION
                        "low", "medium", "high". High uses all available CPUs
                        OR integer percent 1-200
  --data-directory DATA_DIRECTORY
                        Directory Used to save data. default is "./data/"
  -v, --verbose         increase logging verbosity; repeat as -vv for debug and
                        -vvv for trace
```

--------------------------------

### Install and Verify histdatacom

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/data-quality/issue-240-storage-backed-campaign-plan.md

Sets up a Python virtual environment, installs the histdatacom package, and verifies the installation using `runtime doctor`. Ensure `platform.executable_bundled` is true.

```sh
python -m venv .campaign-venv
. .campaign-venv/bin/activate
pip install histdatacom
histdatacom runtime doctor --json
```

--------------------------------

### Verify Production PyPI Install

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

After a successful production publish to PyPI, use this command to verify the installation path. This ensures the package is correctly available and installable from the main PyPI repository.

```sh
bash pypi.sh pypi_install
```

--------------------------------

### Create Project Directory and Navigate (MacOS/Linux)

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Creates a new directory for your project and changes the current directory to it. Use this to start a new project.

```bash
mkdir myproject && cd myproject && pwd
```

--------------------------------

### Start Runtime with Explicit Temporal Executable

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/data-quality/issue-235-batch-3-campaign-report.md

Starts the runtime explicitly using a specified Temporal executable. This is necessary when the source checkout does not bundle the platform executable.

```sh
venv/bin/histdatacom runtime start --executable /opt/local/bin/temporal
```

--------------------------------

### Install histdatacom with Optional Jupyter Support

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install histdatacom with optional notebook support.

```sh
pip install "histdatacom[jupyter]"
```

--------------------------------

### Run Routed Commands with YAML Config

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/configuration.md

Examples of how to execute routed commands using a specified YAML configuration file.

```sh
histdatacom --config recurrent.yaml analytics
histdatacom cleanup --config recurrent.yaml
histdatacom jobs --config recurrent.yaml
histdatacom runtime --config recurrent.yaml
```

--------------------------------

### Install and Use vprof for Profiling

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/perf_snippets.md

Install vprof using pip. Profile a Python script by running vprof with the 'cpmh' command and the script with its arguments.

```bash
$ pip install vprof
```

```bash
$ vprof -c cpmh "src/histdatacom/histdata_com.py -X -p eurusd -f ascii -t tick-data-quotes -s 2022-01 -e 2022-02"
```

--------------------------------

### Setup for Hermetic Testing

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/samples/notebooks/api_quickstart.ipynb

This snippet sets up a hermetic testing environment by installing fake orchestration if the HISTDATACOM_SAMPLE_MODE environment variable is set to 'hermetic'. This is useful for running tests without live API services.

```python
import os

sample_context = None
if os.environ.get("HISTDATACOM_SAMPLE_MODE") == "hermetic":
    from samples._testing import install_fake_orchestration

    sample_context = install_fake_orchestration()
```

--------------------------------

### Example Usage of test_for_cache_or_create

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/api-reference/api-class.md

Demonstrates how to use test_for_cache_or_create to ensure a cache file exists or is created.

```python
from histdatacom.api import Api
from histdatacom.records import Record

record = Record(pair="eurusd", format="ascii", timeframe="M1", ...)
args = {"data_directory": "data/"}

Api.test_for_cache_or_create(record, args)
# Cache file created or confirmed at data/ASCII/M1/eurusd.data
```

--------------------------------

### Start Runtime

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-runtime-runbook.md

Starts the HistData.com runtime using the default resolver. This command initiates the Temporal server and worker lanes.

```sh
histdatacom runtime start
```

--------------------------------

### Install Git Hooks (Windows PowerShell)

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install pre-commit Git hooks for local development on Windows.

```powershell
pre-commit install --install-hooks
```

--------------------------------

### Install Editable Development Version

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install the package in editable mode with development dependencies.

```sh
PYTHONNOUSERSITE=1 python -m pip install -e ".[dev]"
```

--------------------------------

### InfluxDB Configuration Example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/api-reference/influx-class.md

Example YAML format for configuring InfluxDB connection details. This configuration can be loaded from influxdb.yaml or set via environment variables.

```yaml
influxdb:
  org: my_org
  bucket: data_bucket
  url: https://influxdb.example.com:8086
  token: my_influx_token
```

--------------------------------

### Configuration Precedence Example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/configuration.md

Demonstrates how CLI arguments override environment variables and configuration files. The example shows that 'gbpusd' from the CLI is used despite 'eurusd' being set in the environment.

```sh
# Config file has pairs: eurusd
# ENV has no override
# CLI specifies gbpusd
# Result: uses gbpusd

export HISTDATA_PAIRS=eurusd
histdatacom --config config.yaml -p gbpusd
```

--------------------------------

### Install Jupyter Dependency

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/REFERENCE_INDEX.md

Install the Jupyter optional dependency for notebook support.

```sh
pip install histdatacom[jupyter]
```

--------------------------------

### Create Project Directory and Navigate (Windows PowerShell)

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Creates a new directory named 'myproject' and changes the current directory to it. Use this to start a new project on Windows.

```powershell
New-Item -Path "." -Name "myproject" -ItemType "directory"; Set-Location .\myproject\
```

--------------------------------

### Start Runtime with Advanced Configuration

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-runtime-runbook.md

Starts the HistData.com runtime with advanced configuration options, including Temporal executable path, namespace, task queue prefix, CPU utilization, and network multiplier.

```sh
histdatacom runtime start \
  --executable /path/to/temporal \
  --namespace default \
  --task-queue-prefix histdatacom \
  --cpu-utilization medium \
  --network-multiplier 3
```

--------------------------------

### Start Runtime with Executable Override

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-runtime-runbook.md

Starts the HistData.com runtime, allowing you to specify an explicit path to the Temporal executable.

```sh
histdatacom runtime start --executable /path/to/temporal
```

--------------------------------

### Example: Selecting Data Formats

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/types.md

Demonstrates how to specify the desired data formats for API calls using the Format enumeration. Ensure the formats you select are supported.

```python
from histdatacom import Format

# Available formats for API calls
options.formats = {"ascii", "metatrader"}
```

--------------------------------

### Install histdatacom with Optional Pandas Support

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install histdatacom with optional pandas API return formats.

```sh
pip install "histdatacom[pandas]"
```

--------------------------------

### Install histdatacom with Optional Arrow Support

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install histdatacom with optional arrow API return formats.

```sh
pip install "histdatacom[arrow]"
```

--------------------------------

### Install pyarrow Dependency

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/REFERENCE_INDEX.md

Install the pyarrow optional dependency to enable the pyarrow.Table return type.

```sh
pip install histdatacom[arrow]
```

--------------------------------

### YAML Routed Command Sections Configuration

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/configuration.md

This example shows how to configure specific subcommands like analytics, cleanup, jobs, and runtime within a single YAML file.

```yaml
histdatacom:
  # Main configuration
  download_data_archives: true
  pairs:
    - eurusd
  
  # Analytics configuration
  analytics:
    command: feed-regimes
    target: data/ASCII/T/eurusd
    bucket: month
    report: reports/eurusd-feed-regimes.json
    json: true
  
  # Cleanup configuration
  cleanup:
    command: status
    pair_groups:
      - majors
    json: true
  
  # Jobs configuration
  jobs:
    command: list
    offline: true
    json: true
    limit: 20
  
  # Runtime configuration
  runtime:
    command: status
    json: true
```

--------------------------------

### Example: Selecting Pair Groups

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/types.md

Demonstrates how to select predefined groups of currency pairs for data retrieval. This simplifies the process of selecting multiple related pairs.

```python
from histdatacom import Options

options = Options()
options.pair_groups = {"majors", "metals"}  # Select two groups
```

--------------------------------

### Install Optional Dependencies for histdata.com

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/README.md

Install optional dependencies to enable specific features like pandas DataFrame support, pyarrow Table support, InfluxDB imports, or notebook integration. Use 'all' to install all optional features.

```sh
pip install "histdatacom[pandas]"    # pandas.DataFrame support
```

```sh
pip install "histdatacom[arrow]"     # pyarrow.Table support
```

```sh
pip install "histdatacom[influx]"    # InfluxDB import
```

```sh
pip install "histdatacom[jupyter]"   # Notebook support
```

```sh
pip install "histdatacom[all]"       # All optional features
```

--------------------------------

### Install pandas Dependency

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/REFERENCE_INDEX.md

Install the pandas optional dependency to enable the pandas.DataFrame return type.

```sh
pip install histdatacom[pandas]
```

--------------------------------

### Install Temporal Python SDK

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-operations.md

Install the Temporal Python SDK using pip. This is included by default with the histdatacom package.

```sh
pip install histdatacom
```

--------------------------------

### Failure Info Structure Example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/errors.md

An example of the FailureInfo structure, demonstrating error reporting with details and retry policy. Sensitive keys like 'token' are redacted.

```json
{
  "code": "INFLUX_AUTH_FAILED",
  "category": "influx",
  "message": "InfluxDB authentication failed",
  "details": {
    "url": "https://influxdb.example.com:8086",
    "token": "[REDACTED]"
  },
  "location": "influx.py:connect()",
  "retry_policy": "standard"
}
```

--------------------------------

### YAML Configuration with Pair Groups

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/configuration.md

This example demonstrates using `instrument_groups` in the YAML configuration as an alternative to listing individual pairs.

```yaml
histdatacom:
  download_data_archives: true
  extract_csvs: true
  instrument_groups:
    - majors
    - metals
  formats:
    - ascii
  timeframes:
    - tick-data-quotes
  start_yearmonth: 2022-01
  end_yearmonth: 2022-12
```

--------------------------------

### Verify Bundled Platform Wheel Installation

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

Use these commands to inspect a wheel and perform smoke tests after installing a bundled platform wheel. Ensure the wheel is correctly packaged and the runtime components function as expected.

```sh
python scripts/inspect_wheel.py \
  --wheel dist/histdatacom-*-py3-none-<platform-tag>.whl \
  --require-bundled-platform <platform-key>
```

```sh
python scripts/smoke_runtime_install.py \
  --wheel dist/histdatacom-*-py3-none-<platform-tag>.whl \
  --require-bundled-current-platform \
  --check-executable-version \
  --expect-temporal-extra \
  --start-runtime \
  --quality-runtime-smoke
```

--------------------------------

### Create Directory and Navigate (Anaconda Windows)

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Creates a new directory named 'myproject' and changes the current directory to it. Use this to start a new project on Windows with Anaconda.

```shell
mkdir myproject && cd myproject && echo %cd%
```

--------------------------------

### Install and Use VizTracer for Multiprocess Tracing

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/perf_snippets.md

Install VizTracer using pip. Run your Python module with viztracer, specifying tracer entries and command-line arguments. View the results using vizviewer, with options for flame graphs.

```bash
$ pip install viztracer
```

```bash
$ viztracer --tracer_entries 2500000 -m histdatacom -- -X -p eurusd -f ascii -t tick-data-quotes -s 2022-01 -e 2022-02
```

```bash
$ vizviewer /Users/davidmidlo/projects/histdata_com_tools/result.json
```

```bash
$ vizviewer --flamegraph /Users/davidmidlo/projects/histdata_com_tools/result.json
```

--------------------------------

### Install InfluxDB extra for import

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install the InfluxDB extra package to enable the --import_to_influxdb flag. This is required before importing data.

```sh
pip install "histdatacom[influx]"
```

--------------------------------

### Initialize Options Object

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/configuration.md

Initializes the Options object with default settings. This is the starting point for configuring data retrieval.

```python
from histdatacom import Options

options = Options()  # Initializes with defaults
```

--------------------------------

### Start histdatacom with Temporal Executable

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/data-quality/issue-240-storage-backed-campaign-plan.md

Starts the histdatacom runtime with an explicit Temporal executable path. This is suitable for development but distinct from a packaged platform-wheel proof.

```sh
venv/bin/histdatacom runtime start --executable /path/to/temporal
vend/bin/histdatacom runtime doctor --json
```

--------------------------------

### Configuration File: YAML Example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/REFERENCE_INDEX.md

Example of a YAML configuration file for histdata.com tools. Specifies download, extraction, pairs, formats, timeframes, date ranges, data directory, CPU utilization, and orchestration settings.

```yaml
histdatacom:
  download_data_archives: true
  extract_csvs: true
  pairs:
    - eurusd
    - gbpusd
  formats:
    - ascii
  timeframes:
    - 1-minute-bar-quotes
  start_yearmonth: 2022-01
  end_yearmonth: 2022-12
  data_directory: /data/histdata
  cpu_utilization: medium
  orchestration_start: true
```

--------------------------------

### Install and Use SnakeViz for Profiling

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/perf_snippets.md

Install SnakeViz using pip. Profile your Python script using cProfile and save the stats to a file. Visualize the profiling results with the snakeviz command.

```bash
$ pip install snakeviz
```

```bash
$ python -m cProfile -o output.pstats src/histdatacom/histdata_com.py -X -p eurusd -f ascii -t tick-data-quotes -s 2021-01 -e now
```

```bash
$ snakeviz output.pstats
```

--------------------------------

### Full YAML Configuration Example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/configuration.md

This snippet shows a comprehensive YAML configuration for histdata.com tools, including execution modes, data selection, system settings, and orchestration.

```yaml
histdatacom:
  # Execution mode
  download_data_archives: true
  extract_csvs: true
  
  # Data selection
  pairs:
    - eurusd
    - gbpusd
  formats:
    - ascii
  timeframes:
    - 1-minute-bar-quotes
  start_yearmonth: 2022-01
  end_yearmonth: 2022-12
  
  # System
  data_directory: /data/histdata
  cpu_utilization: medium
  verbosity: 1
  
  # Orchestration
  orchestration_start: true
  orchestration_wait_result: true
```

--------------------------------

### Local TestPyPI Preflight Check

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

Performs a local preflight check by building artifacts and serving them through a local simple index. Verifies installation behavior without uploading to TestPyPI.

```bash
bash pypi.sh testpypi_preflight
```

--------------------------------

### Show Help and Options

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Displays all available command-line options and their descriptions. Use this to understand the full range of functionalities.

```txt
histdatacom -h
```

--------------------------------

### Install histdatacom Package

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/README.md

Install the histdatacom package using pip. Optional dependencies for enhanced functionality can be installed with the package.

```sh
pip install histdatacom
# Optional: pip install "histdatacom[pandas,arrow,influx,jupyter]"
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Set up a project-local virtual environment for local development.

```sh
python -m venv venv
source venv/bin/activate
```

--------------------------------

### Example: Requesting Specific Timeframes

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/types.md

Shows how to request specific timeframes for historical data using the Timeframe enumeration. Note that some timeframes are specific to certain platforms like NinjaTrader.

```python
from histdatacom import Timeframe

# Request 1-minute bars and tick data
options.timeframes = {"M1", "T"}
```

--------------------------------

### CLI Error Output Example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/errors.md

Demonstrates the structured error output from the CLI when an invalid command input is provided. This output helps users identify and fix issues.

```sh
$ histdatacom -p invalid_pair

Error: validation
  Pair code 'invalid_pair' not found in supported pairs
  
  Fix the invalid command input or configuration value and rerun.
```

--------------------------------

### Create and Activate Virtual Environment (Windows PowerShell)

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Set up a project-local virtual environment for local development on Windows using PowerShell.

```powershell
py -m venv venv
.\venv\Scripts\Activate.ps1
```

--------------------------------

### CLI: Run with Configuration File

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/REFERENCE_INDEX.md

Demonstrates how to run the histdata.com CLI tool using a specified configuration file.

```sh
Run with: histdatacom --config config.yaml
```

--------------------------------

### Install InfluxDB Dependency

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/REFERENCE_INDEX.md

Install the InfluxDB optional dependency for InfluxDB import functionality.

```sh
pip install histdatacom[influx]
```

--------------------------------

### Install Latest Development Version

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install the latest development version of histdatacom directly from GitHub.

```sh
pip install git+https://github.com/dmidlo/histdata.com-tools.git
```

--------------------------------

### Configuration and CLI Imports

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/ARCHITECTURE.md

Illustrates the module dependencies for the command-line interface (CLI) and configuration loading, including argument parsing, YAML handling, and option management.

```python
cli.py (ArgParser)
  ↓ imports
  ├─ options.py
  ├─ cli_config.py (YAML loading, routed commands)
  ├─ fx_enums.py
  └─ helper_args.py
```

```python
cli_config.py
  ↓ imports
  ├─ options.py
  └─ data_quality/
  └─ data_analytics/
```

--------------------------------

### File System Layout

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/ARCHITECTURE.md

Illustrates the directory structure for Temporal SDK cache, provenance store, logs, runtime configuration, and local data/configuration files.

```text
~/.histdatacom/
├── temporal/                # Temporal SDK cache
│   └── temporal-<version>   # Downloaded executable
├── manifest-status.sqlite3  # Provenance store
├── logs/
│   └── *.log
└── config/
    └── runtime.json         # Runtime state

./
├── data/                    # Default data directory
│   ├── ASCII/
│   │   ├── M1/
│   │   ├── T/
│   │   └── ...
│   ├── MetaTrader/
│   └── .repo                # Repository metadata
├── influxdb.yaml            # InfluxDB credentials (local)
├── config.yaml              # YAML configuration (local)
└── reports/
    ├── quality.json
    └── feed-regimes.json
```

--------------------------------

### Install Editable Development Version (Windows PowerShell)

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Install the package in editable mode with development dependencies on Windows.

```powershell
$env:PYTHONNOUSERSITE = "1"
python -m pip install -e ".[dev]"
```

--------------------------------

### Run Gource

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/perf_snippets.md

Gource is a software visualization tool. Run the command 'gource' to launch it.

```bash
gource
```

--------------------------------

### Handle OrchestrationUnavailableError

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/errors.md

Catch this exception when the Temporal orchestration runtime cannot be started or contacted. It suggests starting the orchestration service.

```python
from histdatacom.orchestration.client import OrchestrationUnavailableError

try:
    histdatacom(options)
except OrchestrationUnavailableError:
    print("Start orchestration with: histdatacom runtime start")
```

--------------------------------

### Example: Running Data Quality Assessment

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/api-reference/data-quality.md

Demonstrates how to discover targets, retrieve quality rules for specific groups, and run the assessment. Prints the summary status and finding count from the generated report.

```python
from histdatacom.data_quality.engine import run_quality_assessment
from histdatacom.data_quality.discovery import discover_quality_targets
from histdatacom.data_quality.rules import get_quality_rules_for_groups

targets = discover_quality_targets(("data/",))
rules = get_quality_rules_for_groups({"ingestion", "time"})

report = run_quality_assessment(targets, rules)
print(f"Status: {report.summary.status}")
print(f"Findings: {report.summary.finding_count}")
```

--------------------------------

### Handle InfluxDependencyError

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/errors.md

This exception is raised when the 'influxdb-client' Python package is not installed. The code snippet provides instructions on how to install the necessary dependency.

```python
from histdatacom.exceptions import InfluxDependencyError

# Install with: pip install "histdatacom[influx]"
```

--------------------------------

### Build and Check Distributions

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

Build project distributions using the PEP 517 build frontend and check the integrity of the built artifacts. This process includes installing the wheel into a fresh virtual environment for smoke testing.

```python
python -m build
```

```python
python -m twine check dist/*.whl dist/*.tar.gz
```

--------------------------------

### Temporal Executable Provisioning Priority

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/ARCHITECTURE.md

Defines the order of preference for obtaining the Temporal executable. It prioritizes environment variables, then bundled wheels, followed by a user cache, and finally a first-run download.

```text
Priority:
1. HISTDATACOM_TEMPORAL_EXECUTABLE env var
2. Offline/private bundled wheel (separate CI path)
3. Verified per-user cache (~/.histdatacom/temporal/)
4. First-run download (pinned URL + checksum index)
```

--------------------------------

### Sample influxdb.yaml configuration

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

A sample configuration file for InfluxDB import. This file should be placed in the current working directory when using the --import_to_influxdb flag.

```yaml
influxdb:
  org: influx_org
  bucket: data_bucket
  url: influx_server_api_url
  token: influx_user_token
```

--------------------------------

### Data quality assessment report example

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-runtime-runbook.md

An example of a text-based report generated by `histdatacom` after a data quality assessment, showing check status, findings, and file details.

```txt
Data quality assessment
checks: ingestion
status: failed
targets: 1 clean: 0 warning: 0 failed: 1
findings: 2 info: 1 warning: 0 error: 1

Clean files
- none

Warning files
- none

Failed files
- csv: /path/to/data/bad/DAT_ASCII_EURUSD_M1_201202_BAD.csv (findings=2, warnings=0, errors=1)
```

--------------------------------

### Build Distributions Locally

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

Builds the source distributions and wheels for the package locally. This is the first step in the release process.

```bash
bash pypi.sh build
```

--------------------------------

### Submit CLI Work Without Starting Temporal

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-operations.md

Submit a data retrieval task without starting a new Temporal instance. Requires an already running Temporal runtime.

```sh
histdatacom --no-orchestration-start \
  -p eurusd -f ascii -t 1-minute-bar-quotes -s now
```

--------------------------------

### Run Local Release Preflight

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

Execute the local release preflight checks without contacting TestPyPI or requiring keyring credentials. This command builds artifacts, applies file-size gates, and runs various verification scripts.

```bash
bash pypi.sh build
bash pypi.sh testpypi_preflight
bash pypi.sh testpypi
bash pypi.sh pypi
```

--------------------------------

### Get AnalyticsDiscoveryResult Target Count

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/api-reference/data-analytics.md

Returns the total number of analytics targets discovered.

```python
@property
def target_count(self) -> int
```

--------------------------------

### Example of Extracting Single Value

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/api-reference/api-class.md

Demonstrates extracting a single value from a Polars DataFrame using _extract_single_value_from_frame.

```python
from histdatacom.api import Api
import polars as pl

df = pl.DataFrame({"col1": [1, 2, 3], "col2": [4.0, 5.0, 6.0]})
value = Api._extract_single_value_from_frame(df, 1, "col2")
print(value)  # 5.0
```

--------------------------------

### Options Constructor

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/api-reference/options.md

Creates a new Options instance with all default values. This is the primary way to initialize the configuration object.

```APIDOC
## `__init__` Constructor

### Description
Creates a new Options instance with all default values.

### Method
```python
def __init__(self) -> None
```

### Parameters
None

### Request Example
```python
from histdatacom import Options

options = Options()
```

### Response
#### Success Response
- **Options Instance**: A new Options object with default configurations.
```

--------------------------------

### Verify Repo Quality with Specific Columns

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/data-quality/issue-238-batch-6-campaign-report.md

Use this command to verify the repository quality for specified instruments, displaying relevant columns and specifying the data directory.

```sh
venv/bin/histdatacom -A \
  -p usdjpy usdmxn usdnok usdpln usdsek usdsgd usdtry usdzar wtiusd xagusd \
  --repo-quality-columns \
  --data-directory data
```

--------------------------------

### Set API Return Type to Polars

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/configuration.md

Sets the return type for API data to `polars.DataFrame`. This is the default and requires no additional installations.

```python
options.api_return_type = "polars"
# Returns: list[dict] with keys "timeframe", "pair", "records", "data"
```

--------------------------------

### Build Platform Wheel with Local Temporal CLI

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

Build a platform-specific wheel using a locally fetched Temporal CLI executable. Set the HISTDATACOM_RUNTIME_EXECUTABLE and HISTDATACOM_FETCH_REPORT environment variables to point to the downloaded executable and its report.

```bash
HISTDATACOM_RUNTIME_EXECUTABLE=.temporal-cli/macos-arm64/bin/temporal \
HISTDATACOM_FETCH_REPORT=.temporal-cli/macos-arm64/temporal-cli-report.json \
bash pypi.sh build
```

--------------------------------

### Download Specific Formats and Timeframes

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Download 1-minute bar quotes for both metastock and excel formats for USDJPY, starting from the current date.

```sh
histdatacom -p usdjpy -f metastock excel -s now
```

--------------------------------

### Run Live Temporal Runtime Throughput Benchmark

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-performance.md

Execute a benchmark for the live Temporal runtime request matrix. Ensure Temporal executable and workspace are correctly configured.

```sh
HISTDATACOM_LIVE_RUNTIME_THROUGHPUT=1 \
HISTDATACOM_TEMPORAL_EXECUTABLE=/opt/local/bin/temporal \
python scripts/benchmark_runtime_throughput.py \
  --workspace /tmp/histdatacom-issue180-benchmark/workspace \
  --runtime-home /tmp/histdatacom-issue180-benchmark/runtime \
  --data-directory /tmp/histdatacom-issue180-benchmark/data \
  --output /tmp/histdatacom-issue180-benchmark/report.json \
  --temporal-executable /opt/local/bin/temporal
```

--------------------------------

### Create and Activate Python Virtual Environment (MacOS/Linux)

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/README.md

Creates a Python virtual environment named 'venv' and activates it. This isolates project dependencies.

```bash
python -m venv venv && source venv/bin/activate
```

--------------------------------

### Submit a histdatacom job

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/docs/temporal-orchestration-operations.md

Example of submitting a job using the histdatacom CLI. This command initiates a data fetching and processing task.

```sh
histdatacom -p eurusd -f ascii -t 1-minute-bar-quotes -s now
```

--------------------------------

### Query Available Repository Data

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/_autodocs/api-reference/main-api.md

Query the repository to get available data ranges for specified currency pairs. This does not download any data.

```python
import histdatacom
from histdatacom import Options

options = Options()
options.available_remote_data = True
options.pairs = {"eurusd", "gbpusd"}

repo = histdatacom(options)
print(type(repo))  # <class 'dict'>
print(repo["eurusd"])  # {'start': '2003-08', 'end': '2024-12'}
```

--------------------------------

### TestPyPI Dry Run

Source: https://github.com/dmidlo/histdata.com-tools/blob/main/RELEASE.md

Tests the release by uploading to TestPyPI and verifying installation behavior. This step should be performed from the 'dev' branch before a production release.

```bash
bash pypi.sh testpypi
```