### Install CRDS and Dependencies (File Submission) Source: https://github.com/spacetelescope/crds/blob/master/README.rst Installs CRDS and its dependencies for file submission roles. This involves cloning the CRDS repository and running a setup script. It's recommended to do this in a dedicated environment. ```bash git clone https://github.com/spacetelescope/crds.git cd crds ./crds_setup_crds ``` -------------------------------- ### Customize CRDS Environment Setup Source: https://github.com/spacetelescope/crds/blob/master/README.rst Demonstrates how to customize the CRDS environment setup by specifying OS, shell, and install location. This allows for more flexible installation configurations. ```bash ./crds_setup_all Linux csh $HOME/miniconda_crds ``` -------------------------------- ### Install CRDS and Dependencies (Full Environment) Source: https://github.com/spacetelescope/crds/blob/master/README.rst Installs a complete environment including a baseline conda, CRDS, and all its dependencies. This is useful for starting from scratch or setting up a dedicated environment for CRDS development and usage. ```bash git clone https://github.com/spacetelescope/crds.git cd crds ./crds_setup_all # open a new terminal window conda activate crds-env ``` -------------------------------- ### Check CRDS Version Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/installation.rst A command-line instruction to check the installed version of the CRDS client software. This is a basic verification step. ```Shell crds list --version ``` -------------------------------- ### Install Full Featured CRDS with Development Dependencies Source: https://github.com/spacetelescope/crds/blob/master/README.rst Installs the latest CRDS from GitHub with a comprehensive set of dependencies, including those for submission, development, testing, and documentation. This is the most complete installation for users actively developing or testing CRDS. ```bash pip install --upgrade git+https://github.com/spacetelescope/crds.git#egg=crds["submission","dev","test","docs"] ``` -------------------------------- ### CRDS Submission Logging Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/programmatic_interface.rst Illustrates the logging output during the CRDS submission process, showing steps like setup, context resolution, file certification, and conformance checks. ```APIDOC 2019-04-24 12:24:50,823 - CRDS - INFO - =============================== setting up =============================== 2019-04-24 12:24:51,038 - CRDS - INFO - Symbolic context 'jwst-edit' resolves to 'jwst_0511.pmap' 2019-04-24 12:24:51,038 - CRDS - INFO - ######################################## 2019-04-24 12:24:51,038 - CRDS - INFO - Certifying './miri_dark_test.fits' (1/1) as 'FITS' relative to context 'jwst_0511.pmap' 2019-04-24 12:24:55,166 - CRDS - INFO - FITS file 'miri_dark_test.fits' conforms to FITS standards. 2019-04-24 12:24:56,219 - CRDS - WARNING - Missing suggested keyword 'META.MODEL_TYPE [DATAMODL]' ``` -------------------------------- ### CRDS Certify: Usage Examples Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Provides practical examples of how to use the `crds.certify` command with different options for validating FITS files and mapping files. ```bash $ crds certify --comparison-context=hst_0027.pmap --run-fitsverify --dump-provenance ./some_reference.fits ``` ```bash $ crds certify ./hst_acs_darkfile_00250.rmap ``` -------------------------------- ### Install Latest CRDS with Submission and Test Dependencies Source: https://github.com/spacetelescope/crds/blob/master/README.rst Uninstalls any existing CRDS installation and installs the latest version from GitHub, including dependencies for submission and testing roles. This is a more advanced installation for users needing specific CRDS features. ```bash pip uninstall --yes crds pip install --upgrade git+https://github.com/spacetelescope/crds.git#egg=crds["submission","test"] ``` -------------------------------- ### Install JWST Software and Dependencies Source: https://github.com/spacetelescope/crds/blob/master/README.rst Installs the JWST calibration software and related dependencies, including numpy and the latest CRDS from GitHub. Note that this might install a CRDS version that is not current enough for specific testing needs. ```bash pip install --upgrade numpy pip install --upgrade git+https://github.com/spacetelescope/jwst ``` -------------------------------- ### Check CRDS Installation and Configuration Source: https://github.com/spacetelescope/crds/blob/master/README.rst This command checks the current installation status of CRDS, displaying version information, mode, paths, cache status, and the effective context. It's useful for verifying a successful installation and understanding the current CRDS environment. ```bash crds list --status ``` -------------------------------- ### Setup Offsite Use for Roman Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/basic_use.rst Sets up CRDS for offsite Roman calibration by defining CRDS_PATH and CRDS_SERVER_URL environment variables before running the pipeline. ```bash $ export CRDS_SERVER_URL=https://roman-crds.stsci.edu # or similar $ export CRDS_PATH=${HOME}/crds_cache $ strun roman_elp dataset.asdf ``` -------------------------------- ### CRDS Parameter Combinations Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/file_submissions.rst Illustrates how different parameter combinations like DETECTOR, CCDAMP, and CCDGAIN are used to define support for specific data files. The example shows a mapping of date ranges to dark frame fits files for the HRC detector. ```Python DETECTOR CCDAMP CCDGAIN ('HRC', 'A|ABCD|AD|B|BC|C|D', '1.0|2.0|4.0|8.0') : UseAfter({ '1992-01-01 00:00:00' : 'lcb12060j_drk.fits', '2002-03-01 00:00:00' : 'n3o1022cj_drk.fits', '2002-03-18 00:00:00' : 'n3o1022ej_drk.fits', '2002-03-19 00:34:31' : 'n3o1022fj_drk.fits', '2002-03-20 00:34:32' : 'n3o1022hj_drk.fits', ... }) ``` -------------------------------- ### Get System Versions (JSONRPC API) Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Shows how to make a POST request to the CRDS JSONRPC endpoint to retrieve system versions. This example includes a successful response with detailed version information. ```bash $ curl -i -X POST -d '{"jsonrpc": "1.0", "method": "get_system_versions", "params": ["0.6.0noop.dev307","null"], "id": 1} ' https://jwst-crds-dit.stsci.edu/json/ HTTP/1.1 200 OK Date: Wed, 24 Aug 2016 22:33:04 GMT Vary: Cookie X-Frame-Options: SAMEORIGIN Content-Type: application/json-rpc Via: 1.1 jwst-crds-dit.stsci.edu Transfer-Encoding: chunked {"error": null, "jsonrpc": "1.0", "id": 1, "result": {"reftype": "CALVER", "author": "Warren J. Hack", "versions": {"TweakRegStep": "0.1.0", "SubtractImagesStep": null, "RSCD_Step": null, "CubeBuildStep": null, "Extract1dStep": null, "AmiAnalyzeStep": "0.7.0.dev", "Extract2dStep": null, "BackgroundStep": null, "SuperBiasStep": null, "DarkCurrentStep": null, "Combine1dStep": null, "SaturationStep": null, "LinearityStep": null, "DQInitStep": null, "ImprintStep": null, "OutlierDetectionStep": null, "AssignWcsStep": null, "KlipStep": null, "StackRefsStep": null, "TweakregCatalogStep": null, "SourceCatalogStep": null, "PersistenceStep": null, "StraylightStep": null, "IPCStep": null, "FlatFieldStep": null, "ResetStep": null, "RefPixStep": null, "ResampleStep": null, "AmiAverageStep": "0.7.0.dev", "FringeStep": null, "AlignRefsStep": null, "LastFrameStep": null, "JumpStep": null, "EmissionStep": null, "WfsCombineStep": null, "AmiNormalizeStep": "0.7.0.dev", "SkyMatchStep": "0.1.0", "PhotomStep": null, "RampFitStep": null, "HlspStep": null}, "instrument": "SYSTEM", "descrip": "JWST calibration processing step version reference file", "CAL_VER": "0.6.0noop. dev307", "history": "Created by cal_ver_steps version 0.7.0.dev"}} ``` -------------------------------- ### Setup Offsite Use for JWST Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/basic_use.rst Sets up CRDS for offsite JWST calibration by defining CRDS_PATH and CRDS_SERVER_URL environment variables before running the pipeline. ```bash $ export CRDS_SERVER_URL=https://jwst-crds.stsci.edu # or similar $ export CRDS_PATH=${HOME}/crds_cache $ strun calwebb_sloper.cfg dataset.fits ``` -------------------------------- ### Install Fitsverify with Conda Source: https://github.com/spacetelescope/crds/blob/master/README.rst Installs the Fitsverify package using conda from the astroconda channel. This is a C-based package not available via pip. ```bash conda config --add channels http://ssb.stsci.edu/astroconda conda install --yes fitsverify ``` -------------------------------- ### Roman WFI FLAT Reference Mapping Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst An example of a reference mapping file (.rmap) for the Roman WFI FLAT. It includes header metadata and a selector that uses `Match` and `UseAfter` to choose flat field files based on detector and optical element. ```Bash header = { 'classes' : ('Match', 'UseAfter'), 'derived_from' : 'roman_wfi_flat_0002.rmap', 'file_ext' : '.asdf', 'filekind' : 'FLAT', 'filetype' : 'FLAT', 'instrument' : 'WFI', 'ld_tpn' : 'wfi_flat_ld.tpn', 'mapping' : 'REFERENCE', 'name' : 'roman_wfi_flat_0002.rmap', 'observatory' : 'ROMAN', 'parkey' : (('META.INSTRUMENT.DETECTOR', 'META.INSTRUMENT.OPTICAL_ELEMENT'), ('META.OBSERVATION.DATE', 'META.OBSERVATION.TIME')), 'sha1sum' : 'bf0119bfbe1d8e5010eb3bec87bc45d575ae8313', 'suffix' : 'flat', 'text_descr' : 'Flat Field', 'tpn' : 'wfi_flat.tpn', } selector = Match({ ('WFI01', 'F158') : UseAfter({ '2020-01-01 00:00:00' : 'roman_wfi_flat_0001.asdf', }), }) ``` -------------------------------- ### JWST Header Examples for get_best_references Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Provides example JSON structures for the `header` parameter when using the `get_best_references` function for JWST data. It shows how to specify instrument type, detector, and filter using both data model dotted identifiers and equivalent FITS header keywords. ```python {"meta.instrument.type":"fgs", "meta.instrument.detector":"fgs1", "meta.instrument.filter":"any"} ``` ```python {"instrume":"fgs", "detector":"fgs1", "filter":"any"} ``` -------------------------------- ### CRDS Master Program Help Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Displays the usage information for the main 'crds' command-line tool, outlining its structure as a wrapper for various CRDS sub-commands and providing an example of assigning HST best reference files. ```bash $ crds --help usage: crds 'crds' is a wrapper used to invoke various CRDS sub-programs. For example, to assign HST best reference files, do: $ crds bestrefs --files *.fits --update-bestrefs Available commands: list -- print information about CRDS configuration, etc. certify -- check CRDS reference and rules files bestrefs -- assign bestrefs to datasets, regressions, repro sync -- manage local CRDS cache, download rules + references diff -- difference CRDS rules and references rowdiff -- difference reference tables matches -- list matching criteria relative to particular rules checksum -- update rmap checksum query_affected -- download CRDS new reference files affected dataset IDs uniqname -- rename HST files with new CDBS-style names get_synphot -- download synphot references newcontext -- automatically generates updated imaps and pmaps refactor -- automatically modifies rmaps For more detail about individual commands use --help: e.g. crds list --help ``` -------------------------------- ### CRDS JSONRPC get_best_references Example (FITS Parkey) Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Demonstrates calling the 'get_best_references' method with FITS parkey names using a Python-like syntax, showing the request and response. ```python >>> jsonrpc.get_best_references("jwst_0000.pmap", {'INSTRUME':'FGS','DETECTOR':'FGS1', 'FILTER':'ANY'}, null) Requesting -> {"id":"jsonrpc", "params":["jwst_0000.pmap", {"INSTRUME":"FGS", "DETECTOR":"FGS1", "FILTER":"ANY"}, null], "method":"get_best_references", "jsonrpc":"1.0"} Deferred(12, unfired) Got -> {"error": null, "jsonrpc": "1.0", "id": "jsonrpc", "result": {"linearity": "jwst_fgs_linearity_0000.fits", "amplifier": "jwst_fgs_amplifier_0000.fits", "mask": "jwst_fgs_mask_0000.fits"}} ``` -------------------------------- ### ROMAN Pipeline Mapping (.pmap) Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst This Python code provides an example of a ROMAN pipeline mapping file, including its header and selector. The header specifies observatory details and the parkey, while the selector maps the WFI instrument to its corresponding instrument mapping file. ```python header = { 'derived_from' : 'roman_0036.pmap', 'description' : 'Updated parkey to include roman prefix.', 'mapping' : 'PIPELINE', 'name' : 'roman_0037.pmap', 'observatory' : 'ROMAN', 'parkey' : ('ROMAN.META.INSTRUMENT.NAME',), 'sha1sum' : '7765a10e18b1607ba268c62033f38130bb995690', } selector = { 'WFI' : 'roman_wfi_0035.imap', } ``` -------------------------------- ### Setup Offsite Use for HST Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/basic_use.rst Configures CRDS for offsite use by setting the CRDS_SERVER_URL and CRDS_PATH environment variables, and then synchronizes references for specified FITS datasets. ```bash $ export CRDS_SERVER_URL=https://hst-crds.stsci.edu # or similar $ export CRDS_PATH=${HOME}/crds_cache $ crds bestrefs --files dataset*.fits --sync-references=1 --update-bestrefs ``` -------------------------------- ### Roman Header Examples for get_best_references Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Provides example JSON structures for the `header` parameter when using the `get_best_references` function for Roman data. It illustrates how to specify exposure type, detector, and table number using Roman Datamodels dotted identifiers. ```python {"roman.meta.exposure.type":"grism", "roman.meta.instrument.detector":"wfi01", "roman.meta.exposure.ma_table_number":"any"} ``` -------------------------------- ### Array Name Specification Examples Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/certify_constraints.rst Illustrates how array names are specified in CRDS, including references to FITS extensions by number and by (name, ver) pair. ```Python Array names are specified as the bare HDU name in the field, e.g. SCI. These are referenced within expressions as _ARRAY. 1. FITS extensions can also named by number, e.g. EXT1 or EXTENSION1 refers to the data of HDU #1. In constraint expressions these are referred to as e.g. EXT1_ARRAY. 2. FITS extensions can be named by (name, ver), in CRDS this is denoted as __, which corresponds to e.g. ('SCI', 1). In constraint expressions they are referred to as as ___ARRAY, e.g. SCI__1_ARRAY. ``` -------------------------------- ### CRDS JSONRPC get_best_references Example (JWST Parkey) Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Demonstrates calling the 'get_best_references' method with JWST data model parkey names using a Python-like syntax, showing the request and response. ```python >>> jsonrpc.get_best_references("jwst_0000.pmap", {'META.INSTRUMENT.TYPE':'FGS','META.INSTRUMENT.DETECTOR':'FGS1', 'META.INSTRUMENT.FILTER':'ANY'}, null) Requesting -> {"id":"jsonrpc", "params":["jwst_0000.pmap", {"META.INSTRUMENT.TYPE":"FGS", "META.INSTRUMENT.DETECTOR":"FGS1", "META.INSTRUMENT.FILTER":"ANY"}, null], "method":"get_best_references", "jsonrpc":"1.0"} Deferred(14, unfired) Got -> {"error": null, "jsonrpc": "1.0", "id": "jsonrpc", "result": {"linearity": "jwst_fgs_linearity_0000.fits", "amplifier": "jwst_fgs_amplifier_0000.fits", "mask": "jwst_fgs_mask_0000.fits"}} ``` -------------------------------- ### CRDS SelectVersion Selector Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst Shows the SelectVersion operator, which selects a reference file based on a software version and relational expressions, prioritizing versions in sorted order. ```Python selector = SelectVersion({ '<3.1': 'cref_flatfield_65.fits', '<5': 'cref_flatfield_73.fits', 'default': 'cref_flatfield_123.fits', }) ``` -------------------------------- ### CRDS Substitution Parameters Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst Demonstrates the definition and usage of substitution parameters in CRDS rmap headers to manage complex matching rules for reference files based on dataset header values. ```Python selector = Match({ ('UVIS', 'G280_AMPS', '1.5', '1.0', '1.0', 'G280-REF', 'T') : UseAfter({ }), }) 'substitutions' : { 'CCDAMP' : { 'G280_AMPS' : ('ABCD', 'A', 'B', 'C', 'D', 'AC', 'AD', 'BC', 'BD'), }, } ``` -------------------------------- ### CRDS Get Recommendations Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/library_use.rst Retrieves recommendations for best calibration references based on dataset parameters and context. It does not cache files locally. Requires network connectivity and potentially environment variables like CRDS_SERVER_URL and CRDS_PATH. ```python def getrecommendations(parameters, reftypes=None, context=None, ignore_cache=False, observatory="jwst", fast=False): """ getrecommendations() returns the best references for the specified `parameters` and pipeline `context`. Unlike getreferences(), getrecommendations() does not attempt to cache the files locally. parameters { str: str,int,float,bool, ... } `parameters` should be a dictionary-like object mapping best reference matching parameters to their values for this dataset. reftypes [ str, ... ] If `reftypes` is None, return all possible reference types. Otherwise return the reference types specified by `reftypes`. context str Specifies the pipeline context, i.e. specific version (.pmap) of CRDS rules used to do the best references match. If `context` is None, use the latest available context. ignore_cache bool If `ignore_cache` is True, download files from server even if already present. observatory str nominally 'jwst' or 'hst'. fast bool If fast is True, skip verbose output, parameter screening, implicit config update, and bad reference checking. Returns { reftype : bestref_basename, ... } returns a mapping from types requested in `reftypes` to the path for each cached reference file. """ ``` -------------------------------- ### CRDS Warning Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/file_submissions.rst Illustrates a warning message from CRDS indicating an 'equal weight special case' where two different match cases are considered equally good for a given parameter set. ```text CRDS - WARNING - ---------------------------------------- Match case (('DETECTOR', 'FUV'),) is an equal weight special case of (('DETECTOR', 'FUV|NUV'),) For some parameter sets, CRDS interprets both matches as equally good. ``` -------------------------------- ### Roman Pattern Specification Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/reference_conventions.rst Illustrates the pattern specification for Roman using 'P_' prefixed keywords in ASDF and datamodels paths, similar to JWST, with OR-ed values and a required trailing '|'. ```python P_OPTICAL_ELEMENT = F213 | F158 | roman.meta.instrument.p_optical_element = F213 | F158 | ``` -------------------------------- ### CRDS JSONRPC Generic Request Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Demonstrates a generic CRDS service request using the UNIX command line utility 'curl'. It shows how to POST a JSONRPC request with method, parameters, and ID. ```bash $ curl -i -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["jwst"], "id": 1}' https://jwst-crds.stsci.edu/json/ ``` -------------------------------- ### CRDS UseAfter Selector Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst Illustrates the UseAfter selector in CRDS, which matches an ordered sequence of date-time values to corresponding reference filenames, selecting the greatest date-time less than or equal to the dataset's EXPSTART. ```Python selector = Match({ ('HRC',) : UseAfter({ '1991-01-01 00:00:00' : 'j4d1435hj_a2d.fits', '1992-01-01 00:00:00' : 'kcb1734ij_a2d.fits', }), ('WFC',) : UseAfter({ '1991-01-01 00:00:00' : 'kcb1734hj_a2d.fits', '2008-01-01 00:00:00' : 't3n1116mj_a2d.fits', }), }) ``` -------------------------------- ### CRDS Sync Command Examples Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/environment.rst Demonstrates how to use the `crds sync` command to fetch test rules and references. The first command fetches all test rules, while the second (commented out) would fetch specifically listed test references. ```bash # Fetch all the test rules $ crds sync --all # Fetch specifically listed test references ``` -------------------------------- ### Get CRDS Context History Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Fetches the history of CRDS contexts that have been activated as the latest context in the pipeline. The history is returned as a list of tuples, each containing a start date, context name, and a description. ```python client.get_context_history("jwst") # Expected output format: # [('2012-09-06 00:00:00', 'jwst.pmap', 'Bootstrap mappings'), ...] ``` -------------------------------- ### Build Server Documentation Source: https://github.com/spacetelescope/crds/blob/master/documentation/README.rst Builds the server-side HTML documentation for the CRDS project. This command is typically executed from within the documentation directory. ```bash $ make serverhtml ``` -------------------------------- ### Get System Versions (JSONRPC API - Error Example) Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Illustrates how the CRDS JSONRPC API handles invalid input, showing a response where 'result' is null and the 'error' field contains a descriptive message. ```bash $ curl -i -X POST -d '{"jsonrpc": "1.0", "method": "get_system_versions", "params": ["an,invalid(version)","null"], "id": 1}' https://jwst-crds-dit.stsci.edu/json/ HTTP/1.1 200 OK Date: Wed, 24 Aug 2016 22:23:11 GMT Vary: Cookie X-Frame-Options: SAMEORIGIN Content-Type: application/json-rpc Via: 1.1 jwst-crds-dit.stsci.edu Transfer-Encoding: chunked ``` -------------------------------- ### Get Default Context Name Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/library_use.rst The get_default_context() function returns the name of the current pipeline mapping (context). For JWST, this is the Build Context corresponding to the installed DMS and Calibration code. For HST and Roman, it's the Latest Context from the CRDS server or local cache. ```python def get_default_context(): """Return the name of the latest pipeline mapping in use for processing files. Returns ------- pipeline context name e.g. 'hst_0007.pmap' """ ``` -------------------------------- ### CRDS Array Data Constraint Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/certify_constraints.rst An example of how array data constraints are structured using utils.Struct, referencing header and array properties like shape, kind, and data. ```Python utils.Struct( SHAPE = hdu.data.shape, KIND = generic_class, DATA_TYPE = typespec, COLUMN_NAMES = column_names, EXTENSION = i, DATA = hdu.data # XXX the difference between 'A' and 'D' constraints! ) ``` -------------------------------- ### Sync References by Context Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Downloads all references mentioned by specified contexts. ```bash $ crds sync --contexts hst_0001.pmap hst_0002.pmap --fetch-references ``` -------------------------------- ### Remove references to ICD-47 in users guide Source: https://github.com/spacetelescope/crds/blob/master/CHANGES.rst Removes outdated references to ICD-47 (Interface Control Document 47) from the user's guide, updating the documentation to reflect current standards or practices. ```Markdown ## User's Guide Updates Removed all mentions and references to ICD-47 to ensure the documentation is current and accurate. ``` -------------------------------- ### CRDS Matches Command Usage and Examples Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Details the 'crds matches' command for displaying selection criteria of reference files against CRDS contexts. Includes options for specifying contexts, files, output formatting, and dataset matching. ```bash usage: crds matches ... [-h] [--contexts [CONTEXT [CONTEXT ...]]] [--files FILES [FILES ...]] [-b] [-o] [-t] Prints out the selection criteria by which the specified references are matched with respect to a particular context. optional arguments: -h, --help show this help message and exit --contexts [CONTEXT [CONTEXT ...]] Specify a list of CRDS mappings to operate on: .pmap, .imap, or .rmap or date-based specification --range MIN:MAX Operate for pipeline context ids (.pmaps) between and . --all Operate with respect to all known CRDS contexts. --last N Operate with respect to the last N contexts. -i, --ignore-cache Download required files even if they're already in the cache. --files FILES [FILES ...] References for which to dump selection criteria. -b, --brief-paths Don't the instrument and filekind. -o, --omit-parameter-names Hide the parameter names of the selection criteria, just show the values. -t, --tuple-format Print the match info as Python tuples. -d DATASETS [DATASETS ...], --datasets DATASETS [DATASETS ...] Dataset ids for which to dump matching parameters from DADSOPS or equivalent database. -c, --condition-values When dumping dataset parameters, first apply CRDS value conditioning / normalization. -m, --minimize-header When dumping dataset parameters, limit them to matching parameters, not historical bestrefs. -v, --verbose Set log verbosity to True, nominal debug level. --verbosity VERBOSITY Set log verbosity to a specific level: 0..100. -R, --readonly-cache Don't modify the CRDS cache. Not compatible with options which implicitly modify the cache. -V, --version Print the software version and exit. -J, --jwst Force observatory to JWST for determining header conventions. -H, --hst Force observatory to HST for determining header conventions. ``` ```bash $ crds matches --contexts hst_0001.pmap --files lc41311jj_pfl.fits lc41311jj_pfl.fits : ACS PFLTFILE DETECTOR='WFC' CCDAMP='A|ABCD|AC|AD|B|BC|BD|C|D' FILTER1='F625W' FILTER2='POL0V' DATE-OBS='1997-01-01' TIME-OBS='00:00:00' ``` ```bash $ crds matches --contexts hst.pmap --files lc41311jj_pfl.fits --omit-parameter-names --brief-paths lc41311jj_pfl.fits : 'WFC' 'A|ABCD|AC|AD|B|BC|BD|C|D' 'F625W' 'POL0V' '1997-01-01' '00:00:00' ``` ```bash $ crds matches --contexts hst.pmap --files lc41311jj_pfl.fits --tuple-format lc41311jj_pfl.fits : (('OBSERVATORY', 'HST'), ('INSTRUMENT', 'ACS'), ('FILEKIND', 'PFLTFILE'), ('DETECTOR', 'WFC'), ('CCDAMP', 'A|ABCD| AC|AD|B|BC|BD|C|D'), ('FILTER1', 'F625W'), ('FILTER2', 'POL0V'), ('DATE-OBS', '1997-01-01'), ('TIME-OBS', '00:00:00')) ``` ```bash $ crds matches --datasets JBANJOF3Q --minimize-headers --contexts hst_0048.pmap hst_0044.pmap ``` -------------------------------- ### CRDS Uses Command Examples Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Demonstrates how to use the 'crds uses' command to list associated CRDS mapping files (.pmap, .imap) and reference files (.rmap) for given FITS files, with options to print datasets and include used files. ```bash $ crds uses --files n3o1022ij_drk.fits --hst hst.pmap hst_0001.pmap hst_0002.pmap hst_0003.pmap ... hst_0041.pmap hst_acs.imap hst_acs_0001.imap hst_acs_0002.imap hst_acs_0003.imap ... hst_acs_0008.imap hst_acs_darkfile.rmap hst_acs_darkfile_0001.rmap hst_acs_darkfile_0002.rmap hst_acs_darkfile_0003.rmap ... hst_acs_darkfile_0005.rmap ``` ```bash $ crds uses --files n3o1022ij_drk.fits --print-datasets --hst J8BA0HRPQ J8BA0IRTQ J8BA0JRWQ J8BA0KT4Q J8BA0LIJQ ``` ```bash $ crds uses --files @dropped --hst --print-datasets --include-used vb41934lj_bia.fits JA7P21A2Q vb41934lj_bia.fits JA7P21A4Q vb41934lj_bia.fits JA7P21A6Q ``` -------------------------------- ### CRDS rmap Header Configuration Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst An example of a Python dictionary representing an rmap header configuration in CRDS. It includes fields like 'parkey', 'extra_keys', 'reffile_switch', 'rmap_relevance', 'rmap_omit', and 'hooks' which control reference file lookups. ```python header = { ... 'parkey' : (('DETECTOR',), ('DATE-OBS', 'TIME-OBS')), 'extra_keys' : ('XCORNER', 'YCORNER', 'CCDCHIP'), 'reffile_switch' : 'BIASCORR', 'reffile_required' : 'YES', 'rmap_relevance' : '((DETECTOR != "SBC") and (BIASCORR != "OMIT"))', 'rmap_omit' : '((DETECTOR != "SBC") and (BIASCORR != "OMIT")), 'parkey_relevance' : { 'binaxis1' : '(DETECTOR == "UVIS")', 'binaxis2' : '(DETECTOR == "UVIS")', 'ccdgain' : '(DETECTOR == "IR")', 'samp_seq' : '(DETECTOR == "IR")', 'subtype' : '(DETECTOR == "IR")', }, 'hooks' : { 'fallback_header' : 'fallback_header_acs_biasfile_v2', 'precondition_header' : 'precondition_header_acs_biasfile_v2', }, ... } ``` -------------------------------- ### HST COS DEADTAB Reference Mapping Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst An example of a reference mapping file (.rmap) for the HST COS DEADTAB. It includes header information and a selector using `Match` and `UseAfter` to select deadtab files based on detector and observation date. ```Bash header = { 'derived_from' : 'generated from CDBS database 2014-05-09 23:24:57.840119', 'filekind' : 'DEADTAB', 'instrument' : 'COS', 'mapping' : 'REFERENCE', 'name' : 'hst_cos_deadtab_0250.rmap', 'observatory' : 'HST', 'parkey' : (('DETECTOR',), ('DATE-OBS', 'TIME-OBS')), 'reffile_format' : 'TABLE', 'reffile_required' : 'NONE', 'reffile_switch' : 'DEADCORR', 'rmap_relevance' : '(DEADCORR != "OMIT")', 'sha1sum' : 'bde314f1848b67891d6309b30eaa5c95611f86e2', } selector = Match({ ('FUV',) : UseAfter({ '1996-10-01 00:00:00' : 's7g1700gl_dead.fits', }), ('NUV',) : UseAfter({ '1996-10-01 00:00:00' : 's7g1700ql_dead.fits', }), }) ``` -------------------------------- ### CRDS Matches Command Usage Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Demonstrates how to use the 'crds matches' command with different output formatting options, including specifying contexts, files, omitting parameter names, using brief paths, and tuple formatting. ```bash $ crds matches --contexts hst_0001.pmap --files lc41311jj_pfl.fits lc41311jj_pfl.fits : ACS PFLTFILE DETECTOR='WFC' CCDAMP='A|ABCD|AC|AD|B|BC|BD|C|D' FILTER1='F625W' FILTER2='F660N' DATE-OBS='1997-01-01' TIME-OBS='00:00:00' ``` ```bash $ crds matches --contexts hst.pmap --files lc41311jj_pfl.fits --omit-parameter-names --brief-paths lc41311jj_pfl.fits : 'WFC' 'A|ABCD|AC|AD|B|BC|BD|C|D' 'F625W' 'POL0V' '1997-01-01' '00:00:00' ``` ```bash $ crds matches --contexts hst.pmap --files lc41311jj_pfl.fits --tuple-format lc41311jj_pfl.fits : (('OBSERVATORY', 'HST'), ('INSTRUMENT', 'ACS'), ('FILEKIND', 'PFLTFILE'), ('DETECTOR', 'WFC'), ('CCDAMP', 'A|ABCD| AC|AD|B|BC|BD|C|D'), ('FILTER1', 'F625W'), ('FILTER2', 'POL0V'), ('DATE-OBS', '1997-01-01'), ('TIME-OBS', '00:00:00')) ``` -------------------------------- ### CRDS Sync - Observatory and File Specification Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Demonstrates how to use `crds.sync` to download CRDS data, specifying the observatory (HST, JWST, Roman) via flags or environment variables. It shows how to sync specific files and output directories. ```bash export CRDS_SERVER_URL=https://hst-crds.stsci.edu crds sync --files 2cd1621ei_dkc.fits --output-dir crds.sync --hst --files 2cd1621ei_dkc.fits --output-dir ``` -------------------------------- ### JWST MIRI DARK Reference Mapping Example Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/rmap_syntax.rst An example of a reference mapping file (.rmap) for the JWST MIRI DARK. It specifies header details and uses a `Match` operator with `UseAfter` to select dark frames based on detector, filter, and read pattern. ```Bash header = { 'derived_from' : 'cloning tool 0.03b (2012-09-11)', 'filekind' : 'DARK', 'instrument' : 'MIRI', 'mapping' : 'REFERENCE', 'name' : 'jwst_miri_dark_0000.rmap', 'observatory' : 'JWST', 'parkey' : (('META.INSTRUMENT.DETECTOR', 'META.INSTRUMENT.FILTER', 'META.EXPOSURE.READPATT'),), 'sha1sum' : '2535d3be806c6e7f5f0da1f2dce64034f9028ddc', } selector = Match({ ('MIRIFULONG', 'ANY', 'FAST') : 'jwst_miri_dark_0000.fits', ('MIRIFULONG', 'ANY', 'SLOW') : 'jwst_miri_dark_0001.fits', ('MIRIFUSHORT', 'ANY', 'FAST') : 'jwst_miri_dark_0002.fits', ('MIRIFUSHORT', 'ANY', 'SLOW') : 'jwst_miri_dark_0003.fits', ('MIRIMAGE', 'ANY', 'FAST') : 'jwst_miri_dark_0004.fits', ('MIRIMAGE', 'ANY', 'SLOW') : 'jwst_miri_dark_0005.fits', }) ``` -------------------------------- ### CRDS Parameter Loading and Saving Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/command_line_tools.rst Demonstrates how to load and save CRDS parameters using pickle (.pkl) or JSON (.json) files. `--load-pickles` loads parameters from a sequence of files, and `--save-pickle` saves the combined results to a specified file format. ```bash crds.bestrefs --load-pickles --save-pickle ``` -------------------------------- ### CRDS JSONRPC Sample URLs Source: https://github.com/spacetelescope/crds/blob/master/documentation/crds_users_guide/source/web_services.rst Provides sample base URLs for making CRDS JSONRPC method calls for different telescopes like HST, JWST, and ROMAN. ```bash http://hst-crds.stsci.edu/json/ ``` ```bash http://jwst-crds.stsci.edu/json/ ``` ```bash http://roman-crds.stsci.edu/json/ ```