### Install sotodlib

Source: https://sotodlib.readthedocs.io/en/latest/_sources/intro.rst.txt

Commands for standard installation or editable development mode installation.

```bash
%> pip install sotodlib
```

```bash
%> cd sotodlib
  %> pip install -e .
```

--------------------------------

### Install TOAST in Development Mode

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Activate the development environment and install the package.

```bash
cd toast
conda activate simons
./platforms/conda.sh
```

--------------------------------

### Install sotodlib

Source: https://sotodlib.readthedocs.io/en/latest/intro.html

Standard installation command for the sotodlib package.

```bash
%> pip install sotodlib
```

--------------------------------

### Load an example DetDb

Source: https://sotodlib.readthedocs.io/en/latest/_sources/context.rst.txt

Retrieve a pre-populated example database for testing and demonstration purposes.

```python
>>> from sotodlib.core import metadata
>>> my_db = metadata.get_example('DetDb')
Creating table base
Creating table geometry
Creating LF-type arrays...
Creating MF-type arrays...
Creating HF-type arrays...
Committing 17094 detectors...
Checking the work...
>>> my_db
<sotodlib.core.metadata.detdb.DetDb object at 0x7f691ccb4080>
```

--------------------------------

### Install Dependencies

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Commands to install required packages so3g and sotodlib.

```bash
pip install so3g
```

```bash
pip install git+https://github.com/simonsobs/sotodlib.git@master
```

```bash
cd sotodlib
pip install .
```

--------------------------------

### Example YAML configuration

Source: https://sotodlib.readthedocs.io/en/latest/_sources/hwp.rst.txt

Annotated example of the YAML configuration file required for G3tHWP.

```yaml
# path to L2 HK directory
data_dir: '/so/level2-daq/satp1/hk/'

# path to ManifestDb directory
output_dir: '/so/metadata/satp1/manifests/hwp_angles/'

# prefix of field name
# for 1st encoder
field_instance: 'satp1.hwp-bbb-e1.feeds.HWPEncoder'

# for 2nd encoder
field_instance_sub: 'satp1.hwp-bbb-e2.feeds.HWPEncoder'

# name list of HWP_encoder field variable
field_list:
- 'rising_edge_count'
- 'irig_time'
- 'counter'
- 'counter_index'
- 'irig_synch_pulse_clock_time'
- 'irig_synch_pulse_clock_counts'
- 'quad'

# HWP OCS packet size
packet_size: 120

# IRIG type
# 0: 1Hz IRIG (default), 1: 10Hz IRIG
irig_type: 0

# number of slots * 2
num_edges: 1140

# Number of slots representing the width of the reference slot
ref_edges: 2

# Threshoild for outlier data to calculate nominal slit width
slit_width_lim: 0.1

# Search range of reference slot
ref_range: 0.1
```

--------------------------------

### Developer Environment Setup

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Commands for setting up a development environment for TOAST.

```bash
conda env list
```

```bash
git clone https://github.com/hpc4cmb/toast.git
cd toast
git checkout -b toast3 -t origin/toast3
```

```bash
cd toast
./platforms/conda_dev_setup.sh simons
```

```bash
cd toast
conda activate simons
./platforms/conda.sh
```

--------------------------------

### Install sotodlib in develop mode

Source: https://sotodlib.readthedocs.io/en/latest/intro.html

Commands to install the package with symlinks for active development.

```bash
%> cd sotodlib
%> pip install -e .
```

--------------------------------

### SourceFlags Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example YAML configuration block for the SourceFlags preprocessing step.

```yaml
- name : "source_flags"
  source_flags_name: "my_source_flags"
  calc:
    mask: {'shape': 'circle',
           'xyr': [0, 0, 1.]}
    center_on: ['jupiter', 'moon'] # list of str
    res: 20 # arcmin
    max_pix: 4000000 # max number of allowed pixels in map
    distance: 0 # max distance of footprint from source in degrees
  save: True
  select: True # optional
    select_source: 'jupiter' # list of str or str. If not provided, all sources from center_on are selected.
    kind: 'any' # 'any', 'all', or float (0.0 < kind < 1.0)
    invert: False # optional, if True logic is filipped.
    Examples:
        1. invert=False, kind='any' → Select detectors with **no** True flags (e.g., for Moon cut).
        2. invert=True, kind='any' → Select detectors with **any** True flags (e.g., for planet selection).
        3. invert=False, kind=0.4 → Select detectors with <40% of True flags.
```

--------------------------------

### Calibrate Configuration Examples

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration blocks for the Calibrate class, demonstrating single value and array-based calibration.

```yaml
  - name: "calibrate"
    process:
      kind: "single_value"
      divide: True # If true will divide instead of multiply.
      # phase_to_pA: 9e6/(2*np.pi)
      val: 1432394.4878270582
  - name: "calibrate"
    process:
      kind: "array"
      cal_array: "cal.array"
    select:
      cut_array: "cal.missing_cal" # should be 0 where cal is good 1 where missing.
```

--------------------------------

### NoiseRatio Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example YAML configuration block for the NoiseRatio preprocessing class.

```yaml
- name: "noise_ratio"
  psd: "psdQ"
  wrap: "noise_ratio_Q"
  subscan: False
  calc:
    f_sel: [0.04, 0.14]
    f_wn: [0.6, 1.0]
  save: True
  select:
    r_max: 1.19
    select_per_detector: True
```

--------------------------------

### Module imports and setup

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/coords/helpers.html

Initial imports and conditional quaternion class assignment.

```python
import so3g.proj
import numpy as np
from pixell import enmap, wcsutils, utils
from . import healpix_utils as hp_utils

import time
import re

DEG = np.pi/180

if hasattr(so3g.proj.quat, "quat"):
    g3quat = so3g.proj.quat.quat
else:
    g3quat = so3g.proj.quat.Quat
```

--------------------------------

### Install TOAST via Conda or Pip

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Standard installation commands for stable releases of TOAST.

```bash
conda install toast
```

```bash
pip install toast
```

--------------------------------

### Run unit tests

Source: https://sotodlib.readthedocs.io/en/latest/_sources/intro.rst.txt

Command to execute the test suite after installation.

```bash
%> python setup.py test
```

--------------------------------

### Configure GetStats

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the GetStats preprocessing step.

```yaml
- name : "tod_stats"
  signal: "signal"  # optional
  wrap: "tod_stats" # optional
  calc:
    stat_names: ["median", "std"]
    split_subscans: False  # optional
    psd_mask:  # optional, for cutting a power spectrum in frequency
      freqs: "psd.freqs"
      low_f: 1
      high_f: 10
  save: True
```

--------------------------------

### Configure GetTauHWP Process

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the GetTauHWP preprocessing step.

```yaml
- name: "get_tau_hwp"
  calc:
    width: 1000
    apodize_samps: 2000
    trim_samps: 2000
    min_fhwp: 1
    max_fhwp: 2
    demod_mode: 4
    name: "tau_hwp"
    merge: False
  save: True
```

--------------------------------

### Get Common Mode Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration entry for the GetCommonMode class.

```yaml
- name: "get_common_mode"
  wrap_name: "common_demodQ"
  calc:
      noise_fit: True
      f_max: 2.0
      signal: "signal"
      method: "median"
  save: True
```

--------------------------------

### Setup Files and Databases

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/update_det_cal.html

Ensures necessary cache files and manifest databases exist, creating them if they are missing.

```python
    def setup_files(self) -> None:
        """Create directories and databases if they don't exist"""
        if not os.path.exists(self.failed_cache_file):
            # If file doesn't exist yet, just create an empty one
            with open(self.failed_cache_file, "w") as f:
                yaml.dump({}, f)

        if not os.path.exists(self.index_path):
            scheme = core.metadata.ManifestScheme()
            scheme.add_exact_match("obs:obs_id")
            scheme.add_data_field("dataset")
            db = core.metadata.ManifestDb(scheme=scheme)
            db.to_file(self.index_path)
```

--------------------------------

### Configure Detector Matching

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

YAML configuration examples for the detector matching pipeline, including a basic setup and a complex SATp1 configuration.

```yaml
results_path: /path/to/results
context_path: /path/to/context.yaml
freq_offset_range_args: None
```

```yaml
results_path: /so/metadata/satp1/manifests/det_match/satp1_det_match_240220m
context_path: /so/metadata/satp1/contexts/smurf_detcal.yaml
show_pb: False
freq_offset_range_args: NULL
apply_solution_pointing: False
solution_type: resonator_set
resonator_set_dir: /so/metadata/satp1/ancillary/detmatch_solutions/satp1_detmatch_solutions_240219r1
match_pars:
  freq_width: 0.2
```

--------------------------------

### Main entry point

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/update_det_cal.html

Initializes the configuration and execution environment to launch the update process.

```python
def main(config_file: str):
    cfg = DetCalCfg.from_yaml(config_file)
    rank, executor, as_completed_callable = get_exec_env(cfg.nprocs_result_set)
    if rank == 0:
        _main(cfg, executor, as_completed_callable)

if __name__ == "__main__":
    main_launcher(main, get_parser)
```

--------------------------------

### Dark Detectors Configuration

Source: https://sotodlib.readthedocs.io/en/latest/preprocess.html

Example configuration block for the DarkDets process.

```yaml
- name : "dark_dets"
  signal: "signal" # optional
  calc: True
  save: True
  select: True
```

--------------------------------

### Setup Development Environment

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Helper script to configure the conda environment for development.

```bash
cd toast
./platforms/conda_dev_setup.sh simons
```

--------------------------------

### Step 1 Configuration for Bright Source Pointing

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

Configuration file example for the first step of the bright source pointing analysis.

```yaml
context_file: /path/to/context.yaml
query_tags: ['moon=1'] #(alternatively specify --sso_name in kwargs)
```

--------------------------------

### EstimateHWPSS Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the EstimateHWPSS class.

```yaml
  - "name : "estimate_hwpss"
    "calc":
      "signal_name": "signal" # optional
      "hwpss_stats_name": "hwpss_stats"
    "save": True
```

--------------------------------

### Configure Make Source Flags

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

Example configuration file format for the make-source-flags tool.

```yaml
# Context for <whatever>
context_file: ./context4_b.yaml

# How to subdivide observations (by detset, but call it "wafer_slot")
subobs:
  use: detset
  label: wafer_slot

# Metadata index & archive filenaming
archive:
  index: 'archive.sqlite'
  policy:
    type: 'simple'
    filename: 'archive.h5'

# Mask parameters
mask_params:
  mask_res: [2, 'arcmin']
  default: {'xyr': [0., 0., 0.1]}
```

--------------------------------

### sotodlib.toast.workflows.setup_load_data_books

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Adds commandline arguments and operators for directly loading L3 books.

```APIDOC
## sotodlib.toast.workflows.setup_load_data_books(operators)

### Description
Add commandline args and operators for directly loading L3 books.

### Parameters
- **operators** (list) - Required - The list of operators to extend.

### Returns
- None
```

--------------------------------

### PCA Filter Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration entry for the PCAFilter class.

```yaml
- name: "pca_filter"
  model_signal: "lpf_signal" # optional, if not provided, use signal
  signal: "signal"
  process:
    n_modes: 10
```

--------------------------------

### Comprehensive Configuration File

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/check_book.html

An advanced YAML configuration example demonstrating various work-arounds, file patterns, and metadata mappings.

```yaml
# Database setup (this is the default).
obsfiledb: './obsfiledb.sqlite'

# For obsdb filenames, path relative to which those names should
# be specified.  (/ is the default.)
root_path: '/'

# Work-arounds
stream_file_pattern: 'D_obs_{stream_id}_{index:03d}.g3'
extra_extra_files: ['frame_splits.txt']
sample_range_inclusive_hack: True
tolerate_missing_ancil: True
tolerate_missing_ancil_timestamps: True
tolerate_timestamps_value_discrepancy: False

# Tolerate arbitrary extra files, except explicitly named ones
tolerate_stray_files: True
banned_files: ['frame_splits.txt']

# If stream_ids are not provided in metadata, list them here.
stream_ids:
  ufm_mv14
  ufm_mv18
  ufm_mv19
  ufm_mv22
  ufm_mv6 
  ufm_mv7 
  ufm_mv9 

# If detset names are not provided in metadata, provide a map from
# stream_id to detset name here.
detset_map:
  ufm_mv14: sch_mv14
  ufm_mv18: sch_mv18
  ufm_mv19: sch_mv19
  ufm_mv22: sch_mv22
  ufm_mv6:  sch_mv6
  ufm_mv7:  sch_mv7
  ufm_mv9:  sch_mv9
```

--------------------------------

### Initialize configuration paths

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/coords/det_match_solutions.html

Sets up paths for wafer maps and focal plane configurations from the environment.

```python
        self.site_pipeline_cfg_dir = os.path.expandvars(self.site_pipeline_cfg_dir)
        if self.wafer_map_path is None:
            self.wafer_map_path = os.path.join(
                self.site_pipeline_cfg_dir, "shared/detmapping/wafer_map.yaml"
            )
        with open(self.wafer_map_path, "r") as f:
            self.wafer_map = yaml.safe_load(f)

        if self.ufm_to_fp_path is None:
            self.ufm_to_fp_path = os.path.join(
                self.site_pipeline_cfg_dir, "shared/focalplane/ufm_to_fp.yaml"
            )
```

--------------------------------

### GlitchFill Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the GlitchFill preprocessing step.

```yaml
  - name: "glitchfill"
    signal: "hwpss_remove"
    flags: "glitches.glitch_flags" # optional
    process:
      nbuf: 10
      use_pca: False
      modes: 1
      in_place: True
      glitch_flags: "glitch_flags"
      wrap: None
```

--------------------------------

### SubtractAzSSTemplate Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the SubtractAzSSTemplate preprocessing step.

```yaml
  - name: "subtract_azss_template"
    process:
      signal: 'signal'
      azss: 'azss_stats_left'
      method: 'interpolate'
      scan_flags: 'left_scan'
      pca_modes: 1
      subtract: True
```

--------------------------------

### Main CLI Entry Point

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/core/metadata/obsfiledb.html

Handles command execution for the so-metadata tool, including file listing and database rerooting logic.

```python
def main(args=None, parser=None):
    """Entry point for the so-metadata tool."""
    if args is None:
        args = sys.argv[1:]
    elif not isinstance(args, argparse.Namespace):
        parser = get_parser()
        args = parser.parse_args(args)

    if args.mode is None:
        args.mode = 'summary'

    db = ObsFileDb.from_file(args.filename, force_new_db=False)

    if args.mode == 'files':
        # Get all files.
        rows = db.conn.execute(
            'select obs_id, name, detset from files '
            'order by obs_id, detset, name').fetchall()

        if args.clean:
            for obs_id, filename, detset in rows:
                print(filename)
        else:
            fmt = '  {obs_id} {detset} {filename}'
            hdr = fmt.format(obs_id="obs_id", detset="detset", filename="Filename")
            print(hdr)
            print('-' * (len(hdr) + 20))
            n = len(rows)
            if n > 20 and not args.all:
                rows = rows[:10]
            for obs_id, filename, detset in rows:
                print(fmt.format(obs_id=obs_id, filename=filename, detset=detset))
            if len(rows) < n:
                print(fmt.format(obs_id='...', filename='+%i others' % (n - len(rows)), detset=''))
                print('(Pass --all to show all results.)')
            print()

    elif args.mode == 'reroot':
        # Reconnect with write?
        if args.overwrite:
            if args.output_db:
                parser.error("Specify only one of --overwrite or --output-db.")
            db = ObsFileDb.from_file(args.filename, force_new_db=True)
            args.output_db = args.filename
        else:
            if args.output_db is None:
                parser.error("Specify an output database name with --output-db, "
                             "or pass --overwrite to clobber.")
            db = ObsFileDb.from_file(args.filename, force_new_db=True)

        # Get all files matching this prefix ...
        c = db.conn.execute('select name from files '
                            'where name like "%s%%"' % (args.old_prefix))
        rows = c.fetchall()
        print('Found %i records matching prefix ...'
               % len(rows))

        print('Converting to new prefix ...')
        n_examples = 1

        if not args.dry_run:
            c = db.conn.cursor()

        for (name, ) in rows:
            new_name = args.new_prefix + name[len(args.old_prefix):]
            if n_examples > 0:
                print(f'  Example: converting filename\n'
                      f'      "{name}"\n'
                      f'    to\n'
                      f'      "{new_name}"')
                n_examples -= 1
            if not args.dry_run:
                c.execute('update files set name=? where name=?', (new_name, name))
```

--------------------------------

### sotodlib.toast.workflows.setup_mapmaker

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Adds commandline arguments, operators, and templates for the TOAST mapmaker.

```APIDOC
## sotodlib.toast.workflows.setup_mapmaker(operators, templates)

### Description
Add commandline args, operators, and templates for TOAST mapmaker.

### Parameters
- **operators** (list) - Required - The list of operators to extend.
- **templates** (list) - Required - The list of templates to extend.
```

--------------------------------

### sotodlib.toast.workflows.setup_job

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Generates configuration for a workflow and parses options, suitable for both batch and interactive use.

```APIDOC
## setup_job(parser=None, operators=[], templates=[], opts=None, config_files=None, argparse_opts=None, verbose=False)

### Description
Generate configuration for a workflow and parse options. This function builds the default configuration and parses overrides from commandline or provided dictionaries.

### Parameters
- **parser** (ArgumentParser) - Optional - The input parser to modify.
- **operators** (list) - Optional - The list of operator instances to configure.
- **templates** (list) - Optional - The list of template instances to configure.
- **opts** (dict) - Optional - The dictionary of interactive config options.
- **config_files** (list) - Optional - A list of configuration files to load.
- **argparse_opts** (list) - Optional - An argparse list of command line arguments to use instead of parsing sys.argv.
- **verbose** (bool) - Optional - If True, dump out the final config to stdout.

### Returns
- **tuple** - The (job namespace, config dictionary, other args, runtime args) from the result of parsing and configuring.
```

--------------------------------

### Configure Demodulation Process

Source: https://sotodlib.readthedocs.io/en/latest/preprocess.html

Example configuration blocks for the Demodulate process, showing standard and HWP-frequency-relative filter settings.

```yaml
- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      bpf_cfg: {'type': 'sine2', 'center': 8, 'width': 3.8, 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': 1.9, 'trans_width': 0.1}
```

```yaml
- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      # You can set float number or str (i.e., ``'4*f_HWP'``) as configs
      bpf_cfg: {'type': 'sine2', 'center': '4*f_HWP', 'width': '3.8*f_HWP', 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': '1.9*f_HWP', 'trans_width': 0.1}
```

--------------------------------

### sotodlib.toast.workflows.setup_noise_estimation

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Adds commandline arguments and operators for estimating detector noise.

```APIDOC
## sotodlib.toast.workflows.setup_noise_estimation(operators)

### Description
Add commandline args and operators for estimating detector noise.

### Parameters
- **operators** (list) - Required - The list of operators to extend.
```

--------------------------------

### PSDCalc Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example YAML configuration block for the PSDCalc preprocessing class.

```yaml
- "name : "psd"
  "signal: "signal" # optional
  "wrap": "psd" # optional
  "process":
    "nperseg": 1024 # optional
    "noverlap": 0 # optional
    "wrap_name": "psd" # optional
    "subscan": False # optional
    "full_output": True # optional
```

--------------------------------

### Trends Preprocessing Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the Trends preprocessing class.

```yaml
- name : "trends"
  signal: "signal" # optional
  calc:
    max_trend: 2.5
    t_piece: 100
  save: True
  plot: True
  select:
    kind: "any"
```

--------------------------------

### Finalize Preprocessing Setup

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/multilayer_preprocess_tod.html

Logs the run list status and initializes the preprocessing databases to avoid race conditions.

```python
    if len(run_list) == 0:
        logger.info("Nothing to run")
        return
    logger.info(f'Run list created with {len(run_list)} obsid groups')

    # ensure dbs exist up front to prevent race conditions
    db_init = pp_util.get_preprocess_db(configs_init, group_by, logger)
    db_proc = pp_util.get_preprocess_db(configs_proc, group_by, logger)

    futures = []
    futures_dict = {}
    obs_errors = {}
```

--------------------------------

### Parallel Execution Environment Setup

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/utils/procs_pool.html

Demonstrates how to refactor a main function to accept an executor and as_completed callable for parallel processing.

```python
>>> def main_func():
>>>     with ProcessPoolExecutor() as executor:
>>>         futures = []
>>>         for i in range(10):
>>>             futures.append(executor.submit(some_func, i))
>>>         for future in as_completed(futures):
>>>             result = future.result()
>>>             print(result)
>>> if __name__ == "__main__":
>>>     main_func()
```

```python
>>> def main_func(executor, as_completed_callable):
>>>     futures = []
>>>     for i in range(10):
>>>         futures.append(executor.submit(some_func, i))
>>>     for future in as_completed_callable(futures):
>>>         result = future.result()
>>>         print(result)
>>> if __name__ == "__main__":
>>>     rank, executor, as_completed_callable = get_exec_env(nprocs=4)
>>>     if rank == 0:
>>>         main_func(executor, as_completed_callable)
```

--------------------------------

### Install optional dependencies

Source: https://sotodlib.readthedocs.io/en/latest/_sources/intro.rst.txt

Syntax for installing specific optional dependency groups via pip.

```bash
%> pip install .[group1,group2]
```

--------------------------------

### G3tHWP.__init__

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/hwp/g3thwp.html

Initializes the G3tHWP manager with an optional configuration file.

```APIDOC
## G3tHWP.__init__(config_file=None)

### Description
Initializes the G3tHWP class to manage L2 HK data into HWP angle g3. If a config file is provided, it loads settings from the YAML file; otherwise, it uses default values.

### Parameters
- **config_file** (str) - Optional - Path to the configuration YAML file.
```

--------------------------------

### Mapmaker Configuration File Example

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

Example YAML configuration file for the maximum likelihood mapmaker.

```yaml
# Query
query: "1"

# Context file containing TODs
context: 'context.yaml'

# Telescope info
freq: 'f150'
site: 'so_lat'

# Mapping area footprint
area: 'geometry.fits'

# Output Directory and file name prefix
odir: './output/'
prefix: 'my_maps'

# Detectors info. null by default
tods: [::100] # Restrict TOD selections by index
ntod: 3 # Special case of `tods` above. Implemented as follows: [:ntod]
nset: 10 # Number of detsets kept
max-dets: 200 # Maximum dets kept
wafer: 'w17' # Restrict which wafers are mapped. Can do multiple wafers

# Mapmaking meta
comps: 'T' # TQU
inject: null 
nocal: True # No relcal or abscal
downsample: 1 # Downsample TOD by this factor
tiled: 0 # Tiling boolean (0 or 1)
nmat-dir: './nmats/' # Dir to save or load nmat
nmat: 'corr' # 'corr' or 'uncorr' 
maxiter: 500 # Max number of iterative steps
nmat_mode: 'build' # 'cache', 'build', 'load' or 'save'
center_at: null 
window: 0.0
inject: null

# Scripting tools
verbose: True
quiet: False
```

--------------------------------

### sotodlib.tod_ops.flags.get_good_distribution_flags

Source: https://sotodlib.readthedocs.io/en/latest/tod_ops.html

Get detector cuts based on the detectors which fit within a gaussian distribution.

```APIDOC
## sotodlib.tod_ops.flags.get_good_distribution_flags

### Description
Get detector cuts based on the detectors which fit within a gaussian distribution.

### Parameters
- **aman** (AxisManager) - Input axis manager.
- **param_name** (str) - Field in aman where to evaluate distribution of.
- **outlier_range** (tuple) - Min and max of distribution allowed.
- **kurtosis_threshold** (float) - Maximum allowed kurtosis of final distribution.
- **blame_max** (bool) - If true, iteratively removes the maximum value.
- **blame_min** (bool) - If true, iteratively removes the minimum value.

### Returns
- **det_mask** (list(bool)) - Boolean mask for cuts True for keep and False for cut.
```

--------------------------------

### Determine rotation start type

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/hwp/g3thwp.html

Identifies the starting pattern of an array based on median differences.

```python
    def _find_rot_start_type(self, arr):
        """ Finds the starting pattern of an array with a bimodal distribution """
        even = np.median(np.diff(arr)[::2][:20])
        odd = np.median(np.diff(arr)[1::2][:20])

        return True if even < odd else False
```

--------------------------------

### Match resonator sets from files

Source: https://sotodlib.readthedocs.io/en/latest/_sources/det_match.rst.txt

Demonstrates loading resonator sets from a tune-file and a solution file, performing the match, and saving the result to an HDF5 file.

```python
from sotodlib.coords import det_match as dm

tunefile = <path_to_tune_file>
bgmap_file = <path_to_bgmap_file>
sol_file = <path_to_solution_file>

src = dm.ResSet.from_tunefile(
    tune_file, north_is_highband=False, bgmap_file=bgmap_file,
    name=f'SAT Tuning'
)
dst = dm.ResSet.from_solutions(
    sol_file, north_is_highband=True, name='solution'
)
match = dm.Match(src, dst)

match.save('match.h5')
```

--------------------------------

### Demodulate Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration for the demodulate process, allowing specification of BPF and LPF configurations.

```yaml
- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      # You can set float number or str (i.e., ``'4*f_HWP'``) as configs
      bpf_cfg: {'type': 'sine2', 'center': '4*f_HWP', 'width': '3.8*f_HWP', 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': '1.9*f_HWP', 'trans_width': 0.1}
```

--------------------------------

### Initialize Preprocess Database

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/preprocess_util.html

Retrieves an existing ManifestDb or creates a new one based on the provided configuration and grouping keys.

```python
def get_preprocess_db(configs, group_by, logger=None):
    """Get or create a ManifestDb found for a given
    config.

    Arguments
    ----------
    configs : dict
        The configuration dictionary.
    group_by : list of str
        The list of keys used to group the detectors.
    logger : PythonLogger
        Optional. Logger object.  If None, a new logger
        is created.

    Returns
    -------
    db : ManifestDb
        ManifestDb object
    """

    if logger is None:
        logger = init_logger("preprocess")

    if os.path.exists(configs['archive']['index']):
        db = core.metadata.ManifestDb(configs['archive']['index'])
    else:
        logger.debug(f"Creating {configs['archive']['index']} as the "
                     "archive index.")
        scheme = core.metadata.ManifestScheme()
        scheme.add_exact_match('obs:obs_id')
        for gb in group_by:
            scheme.add_exact_match('dets:' + gb)
        scheme.add_data_field('dataset')
        db = core.metadata.ManifestDb(
            configs['archive']['index'],
            scheme=scheme
        )
    return db
```

--------------------------------

### Context.__init__

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/core/context.html

Constructs a Context object by loading site, user, and context configuration files and initializing databases.

```APIDOC
## Constructor: Context(filename=None, site_file=None, user_file=None, data=None, load_list='all', metadata_list='all')

### Description
Initializes a new Context object. This object acts as an ordered dictionary containing configuration data and provides access to databases like obsdb, detdb, and obsfiledb.

### Parameters
- **filename** (str) - Optional - Path to the dataset context file.
- **site_file** (str) - Optional - Path to the site file. Defaults to SOTODLIB_SITECONFIG or site.yaml.
- **user_file** (str) - Optional - Path to the user file. Defaults to SOTODLIB_USERCONFIG or ~/.sotodlib.yaml.
- **data** (dict) - Optional - Dictionary of context data to merge into the configuration.
- **load_list** (str or list) - Optional - List of databases to load (e.g., 'obsdb', 'detdb', 'obsfiledb') or 'all'.
- **metadata_list** (str or list) - Optional - List of metadata labels to load or 'all'.
```

--------------------------------

### DetcalNanCuts Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration entry for removing detectors with NaN values in specific calibration fields.

```yaml
- name: "detcal_nan_cuts"
  select:
      fields: [tau_eff, phase_to_pW]
```

--------------------------------

### sotodlib.toast.instrument.Instrument.__init__

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/toast/instrument.html

Initializes the instrument model, either by loading from a hardware file or by simulating a nominal configuration based on telescope parameters.

```APIDOC
## Constructor: Instrument(hw=None, hwfile=None, det_info_file=None, det_info_version=None, telescope=None, sample_rate=10.0, bands=None, wafer_slots=None, tube_slots=None, thinfp=None, creation_time=None, comm=None, apply_net_corr=True)

### Description
Initializes the instrument model. Users can provide an existing Hardware object, a path to a hardware file, or specify telescope parameters to simulate a nominal configuration.

### Parameters
- **hw** (Hardware) - Optional - Existing Hardware object in memory.
- **hwfile** (str) - Optional - Path to a hardware model file on disk.
- **det_info_file** (str) - Optional - Path to a det_info format file.
- **det_info_version** (str) - Optional - Version of the det_info file format.
- **telescope** (str) - Optional - Telescope name to select detectors.
- **sample_rate** (Quantity) - Optional - Sample rate for all detectors (default: 10.0 Hz).
- **bands** (str) - Optional - Comma-separated string of bands to use.
- **wafer_slots** (str) - Optional - Comma-separated string of wafers to use.
- **tube_slots** (str) - Optional - Comma-separated string of tubes to use.
- **thinfp** (int) - Optional - Factor to reduce the number of detectors.
- **creation_time** (float) - Optional - Timestamp for readout_id.
- **comm** (MPI.Comm) - Optional - MPI communicator.
- **apply_net_corr** (bool) - Optional - Whether to degrade NETs according to correlation factors (default: True).
```

--------------------------------

### Atomic Filterbin Map Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

Example YAML configuration file for the atomic filterbin mapmaker, requiring context and preprocess_config parameters.

```yaml
context: /global/cfs/projectdirs/sobs/metadata/satp1/contexts/use_this_local.yaml

# Use a pixell area file for rectangular pixel maps or use an nside value for Healpix maps.
# Only use one of these options
area: band_car_fejer1_5arcmin.fits
#nside: 512

# A query can be a file with a list of obs, or an obsdb query
query: obs_list.txt
#query: "subtype == 'cmb' and timestamp >= 1708743600 and timestamp < 1713672000"

odir: output_directory
preprocess_config: preprocess_config.yaml

# Limit the number of obs, map a specific wafer or band
#ntod: 3
#wafer: ws0
#freq: f090

# Plataform to map
site: so_sat1

# Path to housekeeping data (this is used for extracting pwv)
hk_data_path: /global/cfs/cdirs/sobs/data/site/hk/
```

--------------------------------

### sotodlib.toast.workflows.setup_load_data_hdf5

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Configures the parameters required for loading data from HDF5 files.

```APIDOC
## setup_load_data_hdf5

### Description
Configures the parameters required for loading data from HDF5 files.
```

--------------------------------

### CombineFlags Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the CombineFlags process, demonstrating how to specify flag labels, the output label, and the combination method.

```yaml
- name : "combine_flags"
  process:
    flag_labels: ['glitches.glitch_flags', 'source_flags.jupiter_inv']
    total_flags_label: 'glitch_flags'
    method: 'union' # You can select a method from ['union', '+', 'intersect', '*', 'except', '-'].
    #method: ['+', '*'] # Or you can pass individual method for each flags as a list. 
       # Length of list must match the length of flag_labels.
       # If a list, the first method must be '+', as if adding the first flag set to an empty flag set. 
       # Operations are performed strictly from Left to Right, '*' are not performed first.
```

--------------------------------

### Simulate hardware configuration

Source: https://sotodlib.readthedocs.io/en/latest/cli.html

Generates a mock hardware model and saves it to disk.

```text
usage: so_hardware_sim [options] (use --help for details)

This program simulates the current nominal hardware model and dumps it to disk

optional arguments:
  -h, --help   show this help message and exit
  --out OUT    Name (without extensions) of the output hardware file
  --plain      Write plain text (without gzip compression)
  --overwrite  Overwrite any existing output file.
```

--------------------------------

### FourierFilter Configuration Examples

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Examples of YAML configuration entries for the FourierFilter class, demonstrating single filter, multi-filter, and noise-fit parameter usage.

```yaml
- name: "fourier_filter"
  process:
    filt_function: "timeconst_filter"
    filter_params:
      timeconst: "det_cal.tau_eff"
      invert: True
```

```yaml
- name: "fourier_filter"
      wrap_name: "lpf_demodQ"
      signal_name: "demodQ"
      process:
        filt_function: "sine2"
        filter_params:
          cutoff: 1
          trans_width: 0.1
```

```yaml
- name: "fourier_filter"
  process:
    filters:
      - name: "iir_filter"
        filter_params:
          invert: True
      - name: "timeconst_filter"
        filter_params:
          timeconst: "det_cal.tau_eff"
          invert: True
```

```yaml
- name: "fourier_filter"
  process:
    noise_fit_array: "noiseQ_fit"
    filters:
      - name: "iir_filter"
        filter_params:
          invert: True
      - name: "timeconst_filter"
        filter_params:
          timeconst: "det_cal.tau_eff"
          invert: True
```

--------------------------------

### setup_pointing(operators)

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/toast/workflows/pointing.html

Adds commandline arguments and default pointing operators to the provided list.

```APIDOC
## setup_pointing(operators)

### Description
Adds default pointing operators, including detector pointing, Stokes weights, and various pixelization schemes (Healpix, WCS), to the provided list of operators.

### Parameters
- **operators** (list) - Required - The list of operators to extend.
```

--------------------------------

### sotodlib.vis_hardware.summary_text

Source: https://sotodlib.readthedocs.io/en/latest/_sources/reference.rst.txt

Prints a formatted summary of the hardware configuration to the console.

```APIDOC
## Function: sotodlib.vis_hardware.summary_text(hw)

### Description
Outputs a pretty-formatted summary of the hardware configuration to the console.
```

--------------------------------

### Match.get_stats

Source: https://sotodlib.readthedocs.io/en/latest/det_match.html

Gets stats associated with current matching.

```APIDOC
## get_stats()

### Description
Gets stats associated with current matching.
```

--------------------------------

### SubtractHWPSS Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the SubtractHWPSS class.

```yaml
- name: "subtract_hwpss"
  hwpss_stats: "hwpss_stats"
  process:
    subtract_name: "hwpss_remove"
```

--------------------------------

### G3tHWP Class Initialization

Source: https://sotodlib.readthedocs.io/en/latest/hwp.html

Instantiate the G3tHWP class with a configuration YAML file containing hardware information.

```APIDOC
## G3tHWP(config_file)

### Description
Initializes the G3tHWP analysis tool using a configuration file.

### Parameters
- **config_file** (string) - Required - Path to the YAML configuration file containing HWP hardware settings.
```

--------------------------------

### ObsFileDb Initialization

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/core/metadata/obsfiledb.html

The __init__ method sets up the sqlite3 connection and initializes the database tables if required.

```python
def __init__(self, map_file=None, prefix=None, init_db=True, readonly=False):
        """Instantiate an ObsFileDb.

        Arguments:
          map_file (string): sqlite database file to map.  Defaults to
            ':memory:'.
          prefix (string): as described in class documentation.
          init_db (bool): If True, attempt to create the database
            tables.
          readonly (bool): If True, the database file will be mapped
            in read-only mode.  Not valid on dbs held in :memory:.

        """
        if isinstance(map_file, sqlite3.Connection):
            self.conn = map_file
        else:
            if map_file is None:
                map_file = ':memory:'
            self.conn = sqlite3.connect(map_file)
            uri = False
            if readonly:
                if map_file == ':memory:':
                    raise ValueError('Cannot honor request for readonly db '
                                     'mapped to :memory:.')
                map_file, uri = 'file:%s?mode=ro' % map_file, True

            self.conn = sqlite3.connect(map_file, uri=uri)

        self.conn.row_factory = sqlite3.Row  # access columns by name

        self.prefix = self._get_prefix(map_file, prefix)
        if init_db and not readonly:
            self._create()
```

--------------------------------

### SmurfGapsFlags Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the SmurfGapsFlags preprocessing step.

```yaml
- name: "smurfgaps_flags"
  calc:
    buffer: 200
    name: "smurfgaps"
    merge: True
  save: True
```

--------------------------------

### sotodlib.toast.workflows.setup_load_data_context

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Configures the parameters required for loading data using a context object.

```APIDOC
## setup_load_data_context

### Description
Configures the parameters required for loading data using a context object.

### Parameters
- **args** (object) - Required - The configuration arguments for the workflow.
```

--------------------------------

### AcuDropFlags Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the AcuDropFlags preprocessing step.

```yaml
- name: "acu_drop_flags"
  calc:
    buffer: 200 # disable buffering by setting to False or None.
    name: "acu_drop_flags"
    merge: True
  save: True
```

--------------------------------

### TrimFlagEdge Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the TrimFlagEdge preprocessing step.

```yaml
- name: "trim_flag_edge"
  process:
    flags: "pca_exclude"
```

--------------------------------

### sotodlib.toast.workflows.setup_save_data_books

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Adds commandline arguments and operators for saving to L3 books.

```APIDOC
## sotodlib.toast.workflows.setup_save_data_books(operators)

### Description
Add commandline args and operators for saving to L3 books.

### Parameters
- **operators** (list) - Required - The list of operators to extend.

### Returns
- None
```

--------------------------------

### Rotate QU Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the rotate_qu process.

```yaml
- name : "rotate_qu"
  process:
    sign: 1
    offset: 0
    update_focal_plane: True
```

--------------------------------

### Setup and Simulate Point Source Catalogs

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/toast/workflows/sim_sources.html

Configures and executes the simulation of point source catalogs using Ra/Dec pointing.

```python
def setup_simulate_catalog_signal(operators):
    """Add commandline args and operators for point source catalogs.

    Args:
        operators (list):  The list of operators to extend.

    Returns:
        None

    """
    operators.append(so_ops.SimCatalog(name="sim_catalog", enabled=False))
```

```python
@workflow_timer
def simulate_catalog_signal(job, otherargs, runargs, data):
    """Simulate detector response from a point source catalog.

    Args:
        job (namespace):  The configured operators and templates for this job.
        otherargs (namespace):  Other commandline arguments.
        runargs (namespace):  Job related runtime parameters.
        data (Data):  The data container.

    Returns:
        None

    """
    # Configured operators for this job
    job_ops = job.operators

    if job_ops.sim_catalog.enabled:
        job_ops.sim_catalog.detector_pointing = job_ops.det_pointing_radec_sim
        job_ops.sim_catalog.apply(data)
```

--------------------------------

### PTP Flags Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the ptp_flags process.

```yaml
- name : "ptp_flags"
  calc:
    signal_name: "dsT"
    kurtosis_threshold: 6
  save: True
  select: True
```

--------------------------------

### Filter for Sources Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the filter_for_sources process.

```yaml
- name: "filter_for_sources"
  signal: "signal"
  process:
    n_modes: 10
    source_flags: "source_flags"
    edge_guard: 10 # Number of samples to make the first and last flags False
    trim_samps: 100
    pca_wrap: "pca_model" # optional, if provided, the PCA model is wrapped into aman under this key
```

--------------------------------

### Simulate Hardware Configuration

Source: https://sotodlib.readthedocs.io/en/latest/_sources/cli.rst.txt

Generates a mock hardware configuration file.

```bash
%> so_hardware_sim --out hardware_all --overwrite

Getting example config...
Simulating detectors for telescope LAT...
Simulating detectors for telescope SAT0...
Simulating detectors for telescope SAT1...
Simulating detectors for telescope SAT2...
Simulating detectors for telescope SAT3...
Dumping config to hardware_all.toml.gz...
```

--------------------------------

### Configure CutBadDistribution

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the CutBadDistribution preprocessing step.

```yaml
- name: "cut_bad_dist"
  select:
    param_name: wn_signal
    outlier_range: [0.5, 2.0]
    kurtosis_threshold: 2.0
    blame_max: False
    blame_min: False
```

--------------------------------

### Setup and Load Data Books

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/toast/workflows/data.html

Configures and executes loading data from L3 books.

```python
def setup_load_data_books(operators):
    """Add commandline args and operators for directly loading L3 books.

    Args:
        operators (list):  The list of operators to extend.

    Returns:
        None

    """
    operators.append(so_ops.LoadBooks(name="load_books", enabled=False))
```

```python
@workflow_timer
def load_data_books(job, otherargs, runargs, data):
    """Load data from one or more books.

    Args:
        job (namespace):  The configured operators and templates for this job.
        otherargs (namespace):  Other commandline arguments.
        runargs (namespace):  Job related runtime parameters.
        data (Data):  The data container.

    Returns:
        None

    """
    # Configured operators for this job
    job_ops = job.operators

    # Load it
    if job_ops.load_books.enabled:
        job_ops.load_books.apply(data)
```