SOTODLib

https://sotodlib.readthedocs.io/en/latest
Admin
SOTODLib is a Python library for time-domain data processing and analysis for the Simons Observatory...
Tokens:598,297
Snippets:2,843
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install sotodlib

Source: https://sotodlib.readthedocs.io/en/latest/_sources/intro.rst.txt

Commands for standard installation or editable development mode installation.

```bash
%> pip install sotodlib
```

```bash
%> cd sotodlib
  %> pip install -e .
```

--------------------------------

### Install TOAST in Development Mode

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Activate the development environment and install the package.

```bash
cd toast
conda activate simons
./platforms/conda.sh
```

--------------------------------

### Install sotodlib

Source: https://sotodlib.readthedocs.io/en/latest/intro.html

Standard installation command for the sotodlib package.

```bash
%> pip install sotodlib
```

--------------------------------

### Load an example DetDb

Source: https://sotodlib.readthedocs.io/en/latest/_sources/context.rst.txt

Retrieve a pre-populated example database for testing and demonstration purposes.

```python
>>> from sotodlib.core import metadata
>>> my_db = metadata.get_example('DetDb')
Creating table base
Creating table geometry
Creating LF-type arrays...
Creating MF-type arrays...
Creating HF-type arrays...
Committing 17094 detectors...
Checking the work...
>>> my_db
<sotodlib.core.metadata.detdb.DetDb object at 0x7f691ccb4080>
```

--------------------------------

### Install Dependencies

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Commands to install required packages so3g and sotodlib.

```bash
pip install so3g
```

```bash
pip install git+https://github.com/simonsobs/sotodlib.git@master
```

```bash
cd sotodlib
pip install .
```

--------------------------------

### Example YAML configuration

Source: https://sotodlib.readthedocs.io/en/latest/_sources/hwp.rst.txt

Annotated example of the YAML configuration file required for G3tHWP.

```yaml
# path to L2 HK directory
data_dir: '/so/level2-daq/satp1/hk/'

# path to ManifestDb directory
output_dir: '/so/metadata/satp1/manifests/hwp_angles/'

# prefix of field name
# for 1st encoder
field_instance: 'satp1.hwp-bbb-e1.feeds.HWPEncoder'

# for 2nd encoder
field_instance_sub: 'satp1.hwp-bbb-e2.feeds.HWPEncoder'

# name list of HWP_encoder field variable
field_list:
- 'rising_edge_count'
- 'irig_time'
- 'counter'
- 'counter_index'
- 'irig_synch_pulse_clock_time'
- 'irig_synch_pulse_clock_counts'
- 'quad'

# HWP OCS packet size
packet_size: 120

# IRIG type
# 0: 1Hz IRIG (default), 1: 10Hz IRIG
irig_type: 0

# number of slots * 2
num_edges: 1140

# Number of slots representing the width of the reference slot
ref_edges: 2

# Threshoild for outlier data to calculate nominal slit width
slit_width_lim: 0.1

# Search range of reference slot
ref_range: 0.1
```

--------------------------------

### Developer Environment Setup

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Commands for setting up a development environment for TOAST.

```bash
conda env list
```

```bash
git clone https://github.com/hpc4cmb/toast.git
cd toast
git checkout -b toast3 -t origin/toast3
```

```bash
cd toast
./platforms/conda_dev_setup.sh simons
```

```bash
cd toast
conda activate simons
./platforms/conda.sh
```

--------------------------------

### Install sotodlib in develop mode

Source: https://sotodlib.readthedocs.io/en/latest/intro.html

Commands to install the package with symlinks for active development.

```bash
%> cd sotodlib
%> pip install -e .
```

--------------------------------

### Calibrate Configuration Examples

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration blocks for the Calibrate class, demonstrating single value and array-based calibration.

```yaml
  - name: "calibrate"
    process:
      kind: "single_value"
      divide: True # If true will divide instead of multiply.
      # phase_to_pA: 9e6/(2*np.pi)
      val: 1432394.4878270582
  - name: "calibrate"
    process:
      kind: "array"
      cal_array: "cal.array"
    select:
      cut_array: "cal.missing_cal" # should be 0 where cal is good 1 where missing.
```

--------------------------------

### Configure GetTauHWP Process

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the GetTauHWP preprocessing step.

```yaml
- name: "get_tau_hwp"
  calc:
    width: 1000
    apodize_samps: 2000
    trim_samps: 2000
    min_fhwp: 1
    max_fhwp: 2
    demod_mode: 4
    name: "tau_hwp"
    merge: False
  save: True
```

--------------------------------

### Dark Detectors Configuration

Source: https://sotodlib.readthedocs.io/en/latest/preprocess.html

Example configuration block for the DarkDets process.

```yaml
- name : "dark_dets"
  signal: "signal" # optional
  calc: True
  save: True
  select: True
```

--------------------------------

### Configure Demodulation Process

Source: https://sotodlib.readthedocs.io/en/latest/preprocess.html

Example configuration blocks for the Demodulate process, showing standard and HWP-frequency-relative filter settings.

```yaml
- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      bpf_cfg: {'type': 'sine2', 'center': 8, 'width': 3.8, 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': 1.9, 'trans_width': 0.1}
```

```yaml
- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      # You can set float number or str (i.e., ``'4*f_HWP'``) as configs
      bpf_cfg: {'type': 'sine2', 'center': '4*f_HWP', 'width': '3.8*f_HWP', 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': '1.9*f_HWP', 'trans_width': 0.1}
```

--------------------------------

### Setup Files and Databases

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/update_det_cal.html

Ensures necessary cache files and manifest databases exist, creating them if they are missing.

```python
    def setup_files(self) -> None:
        """Create directories and databases if they don't exist"""
        if not os.path.exists(self.failed_cache_file):
            # If file doesn't exist yet, just create an empty one
            with open(self.failed_cache_file, "w") as f:
                yaml.dump({}, f)

        if not os.path.exists(self.index_path):
            scheme = core.metadata.ManifestScheme()
            scheme.add_exact_match("obs:obs_id")
            scheme.add_data_field("dataset")
            db = core.metadata.ManifestDb(scheme=scheme)
            db.to_file(self.index_path)
```

--------------------------------

### Get Common Mode Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration entry for the GetCommonMode class.

```yaml
- name: "get_common_mode"
  wrap_name: "common_demodQ"
  calc:
      noise_fit: True
      f_max: 2.0
      signal: "signal"
      method: "median"
  save: True
```

--------------------------------

### Main entry point

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/update_det_cal.html

Initializes the configuration and execution environment to launch the update process.

```python
def main(config_file: str):
    cfg = DetCalCfg.from_yaml(config_file)
    rank, executor, as_completed_callable = get_exec_env(cfg.nprocs_result_set)
    if rank == 0:
        _main(cfg, executor, as_completed_callable)

if __name__ == "__main__":
    main_launcher(main, get_parser)
```

--------------------------------

### Configure Detector Matching

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

YAML configuration examples for the detector matching pipeline, including a basic setup and a complex SATp1 configuration.

```yaml
results_path: /path/to/results
context_path: /path/to/context.yaml
freq_offset_range_args: None
```

```yaml
results_path: /so/metadata/satp1/manifests/det_match/satp1_det_match_240220m
context_path: /so/metadata/satp1/contexts/smurf_detcal.yaml
show_pb: False
freq_offset_range_args: NULL
apply_solution_pointing: False
solution_type: resonator_set
resonator_set_dir: /so/metadata/satp1/ancillary/detmatch_solutions/satp1_detmatch_solutions_240219r1
match_pars:
  freq_width: 0.2
```

--------------------------------

### Step 1 Configuration for Bright Source Pointing

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

Configuration file example for the first step of the bright source pointing analysis.

```yaml
context_file: /path/to/context.yaml
query_tags: ['moon=1'] #(alternatively specify --sso_name in kwargs)
```

--------------------------------

### EstimateHWPSS Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the EstimateHWPSS class.

```yaml
  - "name : "estimate_hwpss"
    "calc":
      "signal_name": "signal" # optional
      "hwpss_stats_name": "hwpss_stats"
    "save": True
```

--------------------------------

### sotodlib.tod_ops.flags.get_good_distribution_flags

Source: https://sotodlib.readthedocs.io/en/latest/tod_ops.html

Get detector cuts based on the detectors which fit within a gaussian distribution.

```APIDOC
## sotodlib.tod_ops.flags.get_good_distribution_flags

### Description
Get detector cuts based on the detectors which fit within a gaussian distribution.

### Parameters
- **aman** (AxisManager) - Input axis manager.
- **param_name** (str) - Field in aman where to evaluate distribution of.
- **outlier_range** (tuple) - Min and max of distribution allowed.
- **kurtosis_threshold** (float) - Maximum allowed kurtosis of final distribution.
- **blame_max** (bool) - If true, iteratively removes the maximum value.
- **blame_min** (bool) - If true, iteratively removes the minimum value.

### Returns
- **det_mask** (list(bool)) - Boolean mask for cuts True for keep and False for cut.
```

--------------------------------

### sotodlib.toast.workflows.setup_load_data_books

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Adds commandline arguments and operators for directly loading L3 books.

```APIDOC
## sotodlib.toast.workflows.setup_load_data_books(operators)

### Description
Add commandline args and operators for directly loading L3 books.

### Parameters
- **operators** (list) - Required - The list of operators to extend.

### Returns
- None
```

--------------------------------

### PCA Filter Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration entry for the PCAFilter class.

```yaml
- name: "pca_filter"
  model_signal: "lpf_signal" # optional, if not provided, use signal
  signal: "signal"
  process:
    n_modes: 10
```

--------------------------------

### Comprehensive Configuration File

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/check_book.html

An advanced YAML configuration example demonstrating various work-arounds, file patterns, and metadata mappings.

```yaml
# Database setup (this is the default).
obsfiledb: './obsfiledb.sqlite'

# For obsdb filenames, path relative to which those names should
# be specified.  (/ is the default.)
root_path: '/'

# Work-arounds
stream_file_pattern: 'D_obs_{stream_id}_{index:03d}.g3'
extra_extra_files: ['frame_splits.txt']
sample_range_inclusive_hack: True
tolerate_missing_ancil: True
tolerate_missing_ancil_timestamps: True
tolerate_timestamps_value_discrepancy: False

# Tolerate arbitrary extra files, except explicitly named ones
tolerate_stray_files: True
banned_files: ['frame_splits.txt']

# If stream_ids are not provided in metadata, list them here.
stream_ids:
  ufm_mv14
  ufm_mv18
  ufm_mv19
  ufm_mv22
  ufm_mv6 
  ufm_mv7 
  ufm_mv9 

# If detset names are not provided in metadata, provide a map from
# stream_id to detset name here.
detset_map:
  ufm_mv14: sch_mv14
  ufm_mv18: sch_mv18
  ufm_mv19: sch_mv19
  ufm_mv22: sch_mv22
  ufm_mv6:  sch_mv6
  ufm_mv7:  sch_mv7
  ufm_mv9:  sch_mv9
```

--------------------------------

### GlitchFill Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the GlitchFill preprocessing step.

```yaml
  - name: "glitchfill"
    signal: "hwpss_remove"
    flags: "glitches.glitch_flags" # optional
    process:
      nbuf: 10
      use_pca: False
      modes: 1
      in_place: True
      glitch_flags: "glitch_flags"
      wrap: None
```

--------------------------------

### Initialize configuration paths

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/coords/det_match_solutions.html

Sets up paths for wafer maps and focal plane configurations from the environment.

```python
        self.site_pipeline_cfg_dir = os.path.expandvars(self.site_pipeline_cfg_dir)
        if self.wafer_map_path is None:
            self.wafer_map_path = os.path.join(
                self.site_pipeline_cfg_dir, "shared/detmapping/wafer_map.yaml"
            )
        with open(self.wafer_map_path, "r") as f:
            self.wafer_map = yaml.safe_load(f)

        if self.ufm_to_fp_path is None:
            self.ufm_to_fp_path = os.path.join(
                self.site_pipeline_cfg_dir, "shared/focalplane/ufm_to_fp.yaml"
            )
```

--------------------------------

### Main CLI Entry Point

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/core/metadata/obsfiledb.html

Handles command execution for the so-metadata tool, including file listing and database rerooting logic.

```python
def main(args=None, parser=None):
    """Entry point for the so-metadata tool."""
    if args is None:
        args = sys.argv[1:]
    elif not isinstance(args, argparse.Namespace):
        parser = get_parser()
        args = parser.parse_args(args)

    if args.mode is None:
        args.mode = 'summary'

    db = ObsFileDb.from_file(args.filename, force_new_db=False)

    if args.mode == 'files':
        # Get all files.
        rows = db.conn.execute(
            'select obs_id, name, detset from files '
            'order by obs_id, detset, name').fetchall()

        if args.clean:
            for obs_id, filename, detset in rows:
                print(filename)
        else:
            fmt = '  {obs_id} {detset} {filename}'
            hdr = fmt.format(obs_id="obs_id", detset="detset", filename="Filename")
            print(hdr)
            print('-' * (len(hdr) + 20))
            n = len(rows)
            if n > 20 and not args.all:
                rows = rows[:10]
            for obs_id, filename, detset in rows:
                print(fmt.format(obs_id=obs_id, filename=filename, detset=detset))
            if len(rows) < n:
                print(fmt.format(obs_id='...', filename='+%i others' % (n - len(rows)), detset=''))
                print('(Pass --all to show all results.)')
            print()

    elif args.mode == 'reroot':
        # Reconnect with write?
        if args.overwrite:
            if args.output_db:
                parser.error("Specify only one of --overwrite or --output-db.")
            db = ObsFileDb.from_file(args.filename, force_new_db=True)
            args.output_db = args.filename
        else:
            if args.output_db is None:
                parser.error("Specify an output database name with --output-db, "
                             "or pass --overwrite to clobber.")
            db = ObsFileDb.from_file(args.filename, force_new_db=True)

        # Get all files matching this prefix ...
        c = db.conn.execute('select name from files '
                            'where name like "%s%%"' % (args.old_prefix))
        rows = c.fetchall()
        print('Found %i records matching prefix ...'
               % len(rows))

        print('Converting to new prefix ...')
        n_examples = 1

        if not args.dry_run:
            c = db.conn.cursor()

        for (name, ) in rows:
            new_name = args.new_prefix + name[len(args.old_prefix):]
            if n_examples > 0:
                print(f'  Example: converting filename\n'
                      f'      "{name}"\n'
                      f'    to\n'
                      f'      "{new_name}"')
                n_examples -= 1
            if not args.dry_run:
                c.execute('update files set name=? where name=?', (new_name, name))
```

--------------------------------

### SubtractAzSSTemplate Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the SubtractAzSSTemplate preprocessing step.

```yaml
  - name: "subtract_azss_template"
    process:
      signal: 'signal'
      azss: 'azss_stats_left'
      method: 'interpolate'
      scan_flags: 'left_scan'
      pca_modes: 1
      subtract: True
```

--------------------------------

### sotodlib.toast.workflows.setup_mapmaker

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Adds commandline arguments, operators, and templates for the TOAST mapmaker.

```APIDOC
## sotodlib.toast.workflows.setup_mapmaker(operators, templates)

### Description
Add commandline args, operators, and templates for TOAST mapmaker.

### Parameters
- **operators** (list) - Required - The list of operators to extend.
- **templates** (list) - Required - The list of templates to extend.
```

--------------------------------

### sotodlib.toast.workflows.setup_job

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Generates configuration for a workflow and parses options, suitable for both batch and interactive use.

```APIDOC
## setup_job(parser=None, operators=[], templates=[], opts=None, config_files=None, argparse_opts=None, verbose=False)

### Description
Generate configuration for a workflow and parse options. This function builds the default configuration and parses overrides from commandline or provided dictionaries.

### Parameters
- **parser** (ArgumentParser) - Optional - The input parser to modify.
- **operators** (list) - Optional - The list of operator instances to configure.
- **templates** (list) - Optional - The list of template instances to configure.
- **opts** (dict) - Optional - The dictionary of interactive config options.
- **config_files** (list) - Optional - A list of configuration files to load.
- **argparse_opts** (list) - Optional - An argparse list of command line arguments to use instead of parsing sys.argv.
- **verbose** (bool) - Optional - If True, dump out the final config to stdout.

### Returns
- **tuple** - The (job namespace, config dictionary, other args, runtime args) from the result of parsing and configuring.
```

--------------------------------

### SourceFlags Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example YAML configuration block for the SourceFlags preprocessing step.

```yaml
- name : "source_flags"
  source_flags_name: "my_source_flags"
  calc:
    mask: {'shape': 'circle',
           'xyr': [0, 0, 1.]}
    center_on: ['jupiter', 'moon'] # list of str
    res: 20 # arcmin
    max_pix: 4000000 # max number of allowed pixels in map
    distance: 0 # max distance of footprint from source in degrees
  save: True
  select: True # optional
    select_source: 'jupiter' # list of str or str. If not provided, all sources from center_on are selected.
    kind: 'any' # 'any', 'all', or float (0.0 < kind < 1.0)
    invert: False # optional, if True logic is filipped.
    Examples:
        1. invert=False, kind='any' → Select detectors with **no** True flags (e.g., for Moon cut).
        2. invert=True, kind='any' → Select detectors with **any** True flags (e.g., for planet selection).
        3. invert=False, kind=0.4 → Select detectors with <40% of True flags.
```

--------------------------------

### NoiseRatio Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example YAML configuration block for the NoiseRatio preprocessing class.

```yaml
- name: "noise_ratio"
  psd: "psdQ"
  wrap: "noise_ratio_Q"
  subscan: False
  calc:
    f_sel: [0.04, 0.14]
    f_wn: [0.6, 1.0]
  save: True
  select:
    r_max: 1.19
    select_per_detector: True
```

--------------------------------

### Module imports and setup

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/coords/helpers.html

Initial imports and conditional quaternion class assignment.

```python
import so3g.proj
import numpy as np
from pixell import enmap, wcsutils, utils
from . import healpix_utils as hp_utils

import time
import re

DEG = np.pi/180

if hasattr(so3g.proj.quat, "quat"):
    g3quat = so3g.proj.quat.quat
else:
    g3quat = so3g.proj.quat.Quat
```

--------------------------------

### sotodlib.toast.workflows.setup_noise_estimation

Source: https://sotodlib.readthedocs.io/en/latest/toast.html

Adds commandline arguments and operators for estimating detector noise.

```APIDOC
## sotodlib.toast.workflows.setup_noise_estimation(operators)

### Description
Add commandline args and operators for estimating detector noise.

### Parameters
- **operators** (list) - Required - The list of operators to extend.
```

--------------------------------

### PSDCalc Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example YAML configuration block for the PSDCalc preprocessing class.

```yaml
- "name : "psd"
  "signal: "signal" # optional
  "wrap": "psd" # optional
  "process":
    "nperseg": 1024 # optional
    "noverlap": 0 # optional
    "wrap_name": "psd" # optional
    "subscan": False # optional
    "full_output": True # optional
```

--------------------------------

### Trends Preprocessing Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the Trends preprocessing class.

```yaml
- name : "trends"
  signal: "signal" # optional
  calc:
    max_trend: 2.5
    t_piece: 100
  save: True
  plot: True
  select:
    kind: "any"
```

--------------------------------

### Finalize Preprocessing Setup

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/site_pipeline/multilayer_preprocess_tod.html

Logs the run list status and initializes the preprocessing databases to avoid race conditions.

```python
    if len(run_list) == 0:
        logger.info("Nothing to run")
        return
    logger.info(f'Run list created with {len(run_list)} obsid groups')

    # ensure dbs exist up front to prevent race conditions
    db_init = pp_util.get_preprocess_db(configs_init, group_by, logger)
    db_proc = pp_util.get_preprocess_db(configs_proc, group_by, logger)

    futures = []
    futures_dict = {}
    obs_errors = {}
```

--------------------------------

### Parallel Execution Environment Setup

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/utils/procs_pool.html

Demonstrates how to refactor a main function to accept an executor and as_completed callable for parallel processing.

```python
>>> def main_func():
>>>     with ProcessPoolExecutor() as executor:
>>>         futures = []
>>>         for i in range(10):
>>>             futures.append(executor.submit(some_func, i))
>>>         for future in as_completed(futures):
>>>             result = future.result()
>>>             print(result)
>>> if __name__ == "__main__":
>>>     main_func()
```

```python
>>> def main_func(executor, as_completed_callable):
>>>     futures = []
>>>     for i in range(10):
>>>         futures.append(executor.submit(some_func, i))
>>>     for future in as_completed_callable(futures):
>>>         result = future.result()
>>>         print(result)
>>> if __name__ == "__main__":
>>>     rank, executor, as_completed_callable = get_exec_env(nprocs=4)
>>>     if rank == 0:
>>>         main_func(executor, as_completed_callable)
```

--------------------------------

### Install TOAST via Conda or Pip

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Standard installation commands for stable releases of TOAST.

```bash
conda install toast
```

```bash
pip install toast
```

--------------------------------

### Install optional dependencies

Source: https://sotodlib.readthedocs.io/en/latest/_sources/intro.rst.txt

Syntax for installing specific optional dependency groups via pip.

```bash
%> pip install .[group1,group2]
```

--------------------------------

### Mapmaker Configuration File Example

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

Example YAML configuration file for the maximum likelihood mapmaker.

```yaml
# Query
query: "1"

# Context file containing TODs
context: 'context.yaml'

# Telescope info
freq: 'f150'
site: 'so_lat'

# Mapping area footprint
area: 'geometry.fits'

# Output Directory and file name prefix
odir: './output/'
prefix: 'my_maps'

# Detectors info. null by default
tods: [::100] # Restrict TOD selections by index
ntod: 3 # Special case of `tods` above. Implemented as follows: [:ntod]
nset: 10 # Number of detsets kept
max-dets: 200 # Maximum dets kept
wafer: 'w17' # Restrict which wafers are mapped. Can do multiple wafers

# Mapmaking meta
comps: 'T' # TQU
inject: null 
nocal: True # No relcal or abscal
downsample: 1 # Downsample TOD by this factor
tiled: 0 # Tiling boolean (0 or 1)
nmat-dir: './nmats/' # Dir to save or load nmat
nmat: 'corr' # 'corr' or 'uncorr' 
maxiter: 500 # Max number of iterative steps
nmat_mode: 'build' # 'cache', 'build', 'load' or 'save'
center_at: null 
window: 0.0
inject: null

# Scripting tools
verbose: True
quiet: False
```

--------------------------------

### G3tHWP.__init__

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/hwp/g3thwp.html

Initializes the G3tHWP manager with an optional configuration file.

```APIDOC
## G3tHWP.__init__(config_file=None)

### Description
Initializes the G3tHWP class to manage L2 HK data into HWP angle g3. If a config file is provided, it loads settings from the YAML file; otherwise, it uses default values.

### Parameters
- **config_file** (str) - Optional - Path to the configuration YAML file.
```

--------------------------------

### Determine rotation start type

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/hwp/g3thwp.html

Identifies the starting pattern of an array based on median differences.

```python
    def _find_rot_start_type(self, arr):
        """ Finds the starting pattern of an array with a bimodal distribution """
        even = np.median(np.diff(arr)[::2][:20])
        odd = np.median(np.diff(arr)[1::2][:20])

        return True if even < odd else False
```

--------------------------------

### Match resonator sets from files

Source: https://sotodlib.readthedocs.io/en/latest/_sources/det_match.rst.txt

Demonstrates loading resonator sets from a tune-file and a solution file, performing the match, and saving the result to an HDF5 file.

```python
from sotodlib.coords import det_match as dm

tunefile = <path_to_tune_file>
bgmap_file = <path_to_bgmap_file>
sol_file = <path_to_solution_file>

src = dm.ResSet.from_tunefile(
    tune_file, north_is_highband=False, bgmap_file=bgmap_file,
    name=f'SAT Tuning'
)
dst = dm.ResSet.from_solutions(
    sol_file, north_is_highband=True, name='solution'
)
match = dm.Match(src, dst)

match.save('match.h5')
```

--------------------------------

### Demodulate Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration for the demodulate process, allowing specification of BPF and LPF configurations.

```yaml
- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      # You can set float number or str (i.e., ``'4*f_HWP'``) as configs
      bpf_cfg: {'type': 'sine2', 'center': '4*f_HWP', 'width': '3.8*f_HWP', 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': '1.9*f_HWP', 'trans_width': 0.1}
```

--------------------------------

### Initialize Preprocess Database

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/preprocess_util.html

Retrieves an existing ManifestDb or creates a new one based on the provided configuration and grouping keys.

```python
def get_preprocess_db(configs, group_by, logger=None):
    """Get or create a ManifestDb found for a given
    config.

    Arguments
    ----------
    configs : dict
        The configuration dictionary.
    group_by : list of str
        The list of keys used to group the detectors.
    logger : PythonLogger
        Optional. Logger object.  If None, a new logger
        is created.

    Returns
    -------
    db : ManifestDb
        ManifestDb object
    """

    if logger is None:
        logger = init_logger("preprocess")

    if os.path.exists(configs['archive']['index']):
        db = core.metadata.ManifestDb(configs['archive']['index'])
    else:
        logger.debug(f"Creating {configs['archive']['index']} as the "
                     "archive index.")
        scheme = core.metadata.ManifestScheme()
        scheme.add_exact_match('obs:obs_id')
        for gb in group_by:
            scheme.add_exact_match('dets:' + gb)
        scheme.add_data_field('dataset')
        db = core.metadata.ManifestDb(
            configs['archive']['index'],
            scheme=scheme
        )
    return db
```

--------------------------------

### Context.__init__

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/core/context.html

Constructs a Context object by loading site, user, and context configuration files and initializing databases.

```APIDOC
## Constructor: Context(filename=None, site_file=None, user_file=None, data=None, load_list='all', metadata_list='all')

### Description
Initializes a new Context object. This object acts as an ordered dictionary containing configuration data and provides access to databases like obsdb, detdb, and obsfiledb.

### Parameters
- **filename** (str) - Optional - Path to the dataset context file.
- **site_file** (str) - Optional - Path to the site file. Defaults to SOTODLIB_SITECONFIG or site.yaml.
- **user_file** (str) - Optional - Path to the user file. Defaults to SOTODLIB_USERCONFIG or ~/.sotodlib.yaml.
- **data** (dict) - Optional - Dictionary of context data to merge into the configuration.
- **load_list** (str or list) - Optional - List of databases to load (e.g., 'obsdb', 'detdb', 'obsfiledb') or 'all'.
- **metadata_list** (str or list) - Optional - List of metadata labels to load or 'all'.
```

--------------------------------

### DetcalNanCuts Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration entry for removing detectors with NaN values in specific calibration fields.

```yaml
- name: "detcal_nan_cuts"
  select:
      fields: [tau_eff, phase_to_pW]
```

--------------------------------

### sotodlib.toast.instrument.Instrument.__init__

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/toast/instrument.html

Initializes the instrument model, either by loading from a hardware file or by simulating a nominal configuration based on telescope parameters.

```APIDOC
## Constructor: Instrument(hw=None, hwfile=None, det_info_file=None, det_info_version=None, telescope=None, sample_rate=10.0, bands=None, wafer_slots=None, tube_slots=None, thinfp=None, creation_time=None, comm=None, apply_net_corr=True)

### Description
Initializes the instrument model. Users can provide an existing Hardware object, a path to a hardware file, or specify telescope parameters to simulate a nominal configuration.

### Parameters
- **hw** (Hardware) - Optional - Existing Hardware object in memory.
- **hwfile** (str) - Optional - Path to a hardware model file on disk.
- **det_info_file** (str) - Optional - Path to a det_info format file.
- **det_info_version** (str) - Optional - Version of the det_info file format.
- **telescope** (str) - Optional - Telescope name to select detectors.
- **sample_rate** (Quantity) - Optional - Sample rate for all detectors (default: 10.0 Hz).
- **bands** (str) - Optional - Comma-separated string of bands to use.
- **wafer_slots** (str) - Optional - Comma-separated string of wafers to use.
- **tube_slots** (str) - Optional - Comma-separated string of tubes to use.
- **thinfp** (int) - Optional - Factor to reduce the number of detectors.
- **creation_time** (float) - Optional - Timestamp for readout_id.
- **comm** (MPI.Comm) - Optional - MPI communicator.
- **apply_net_corr** (bool) - Optional - Whether to degrade NETs according to correlation factors (default: True).
```

--------------------------------

### Atomic Filterbin Map Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_sources/site_pipeline.rst.txt

Example YAML configuration file for the atomic filterbin mapmaker, requiring context and preprocess_config parameters.

```yaml
context: /global/cfs/projectdirs/sobs/metadata/satp1/contexts/use_this_local.yaml

# Use a pixell area file for rectangular pixel maps or use an nside value for Healpix maps.
# Only use one of these options
area: band_car_fejer1_5arcmin.fits
#nside: 512

# A query can be a file with a list of obs, or an obsdb query
query: obs_list.txt
#query: "subtype == 'cmb' and timestamp >= 1708743600 and timestamp < 1713672000"

odir: output_directory
preprocess_config: preprocess_config.yaml

# Limit the number of obs, map a specific wafer or band
#ntod: 3
#wafer: ws0
#freq: f090

# Plataform to map
site: so_sat1

# Path to housekeeping data (this is used for extracting pwv)
hk_data_path: /global/cfs/cdirs/sobs/data/site/hk/
```

--------------------------------

### sotodlib.toast.workflows.setup_load_data_hdf5

Source: https://sotodlib.readthedocs.io/en/latest/_sources/toast.rst.txt

Configures the parameters required for loading data from HDF5 files.

```APIDOC
## setup_load_data_hdf5

### Description
Configures the parameters required for loading data from HDF5 files.
```

--------------------------------

### CombineFlags Configuration Example

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Example configuration block for the CombineFlags process, demonstrating how to specify flag labels, the output label, and the combination method.

```yaml
- name : "combine_flags"
  process:
    flag_labels: ['glitches.glitch_flags', 'source_flags.jupiter_inv']
    total_flags_label: 'glitch_flags'
    method: 'union' # You can select a method from ['union', '+', 'intersect', '*', 'except', '-'].
    #method: ['+', '*'] # Or you can pass individual method for each flags as a list. 
       # Length of list must match the length of flag_labels.
       # If a list, the first method must be '+', as if adding the first flag set to an empty flag set. 
       # Operations are performed strictly from Left to Right, '*' are not performed first.
```

--------------------------------

### Simulate hardware configuration

Source: https://sotodlib.readthedocs.io/en/latest/cli.html

Generates a mock hardware model and saves it to disk.

```text
usage: so_hardware_sim [options] (use --help for details)

This program simulates the current nominal hardware model and dumps it to disk

optional arguments:
  -h, --help   show this help message and exit
  --out OUT    Name (without extensions) of the output hardware file
  --plain      Write plain text (without gzip compression)
  --overwrite  Overwrite any existing output file.
```

--------------------------------

### FourierFilter Configuration Examples

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/preprocess/processes.html

Examples of YAML configuration entries for the FourierFilter class, demonstrating single filter, multi-filter, and noise-fit parameter usage.

```yaml
- name: "fourier_filter"
  process:
    filt_function: "timeconst_filter"
    filter_params:
      timeconst: "det_cal.tau_eff"
      invert: True
```

```yaml
- name: "fourier_filter"
      wrap_name: "lpf_demodQ"
      signal_name: "demodQ"
      process:
        filt_function: "sine2"
        filter_params:
          cutoff: 1
          trans_width: 0.1
```

```yaml
- name: "fourier_filter"
  process:
    filters:
      - name: "iir_filter"
        filter_params:
          invert: True
      - name: "timeconst_filter"
        filter_params:
          timeconst: "det_cal.tau_eff"
          invert: True
```

```yaml
- name: "fourier_filter"
  process:
    noise_fit_array: "noiseQ_fit"
    filters:
      - name: "iir_filter"
        filter_params:
          invert: True
      - name: "timeconst_filter"
        filter_params:
          timeconst: "det_cal.tau_eff"
          invert: True
```

--------------------------------

### setup_pointing(operators)

Source: https://sotodlib.readthedocs.io/en/latest/_modules/sotodlib/toast/workflows/pointing.html

Adds commandline arguments and default pointing operators to the provided list.

```APIDOC
## setup_pointing(operators)

### Description
Adds default pointing operators, including detector pointing, Stokes weights, and various pixelization schemes (Healpix, WCS), to the provided list of operators.

### Parameters
- **operators** (list) - Required - The list of operators to extend.
```

--------------------------------

### Match.get_stats

Source: https://sotodlib.readthedocs.io/en/latest/det_match.html

Gets stats associated with current matching.

```APIDOC
## get_stats()

### Description
Gets stats associated with current matching.
```

--------------------------------

### sotodlib.vis_hardware.summary_text

Source: https://sotodlib.readthedocs.io/en/latest/_sources/reference.rst.txt

Prints a formatted summary of the hardware configuration to the console.

```APIDOC
## Function: sotodlib.vis_hardware.summary_text(hw)

### Description
Outputs a pretty-formatted summary of the hardware configuration to the console.
```
Tokens:
Raw