### Execute Installation Script
Source: https://desispec.readthedocs.io/en/latest/install.html
Run the install.sh script to perform the initial installation of DESI software from source. Ensure the script is executable before running.
```bash
%> ./install.sh
```
--------------------------------
### Install CFITSIO, Boost, and OpenMPI with Homebrew
Source: https://desispec.readthedocs.io/en/latest/install.html
Installs core non-Python dependencies using Homebrew on OS X.
```bash
%> brew install cfitsio
%> brew install boost
%> brew install openmpi
```
--------------------------------
### Execute DESI Installation Script
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Run the 'install.sh' script to perform the initial installation of DESI software from source. Ensure the script is executable before running.
```bash
./install.sh
```
--------------------------------
### Configure and Install HARP from Tarball
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Install the HARP package from a release tarball. This example shows configuration options for disabling Python and MPI support, and setting the installation prefix.
```bash
cd harp-1.0.1
./configure --disable-python --disable-mpi \
--prefix="${HOME}/desi-${NERSC_HOST}"
```
--------------------------------
### Fiber Bitmask Examples
Source: https://desispec.readthedocs.io/en/latest/api.html
Examples demonstrating how to define bitmasks using constants, comparison values, or direct integer representation.
```python
bitmask = [fmsk.BROKENFIBER,fmsk.UNASSIGNED,fmsk.BADFIBER,
fmsk.BADTRACE, fmsk.MANYREJECTED]
```
```python
bitmask = get_fiberbitmask_comparison_value(kind='fluxcalib', band='brz')
```
```python
bitmask = 'fluxcalib'
```
```python
bitmask = 4128780
```
--------------------------------
### DESI Software Installation Script
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
A helper script to automate the installation of DESI packages from their git source trees. It cleans, builds, and installs each package to a specified prefix.
```bash
#!/bin/bash
# This should be your actual install location...
pref="${HOME}/desi-${NERSC_HOST}"
cd specex
make clean
SPECEX_PREFIX=${pref} make -j 4 install
cd ..
for pkg in desiutil desimodel desitarget desisim specter desispec; do
cd ${pkg}
python setup.py clean
python setup.py install --prefix=${pref}
cd ..
done
```
--------------------------------
### Install DESI Packages from Source
Source: https://desispec.readthedocs.io/en/latest/install.html
A bash script to clean, build, and install DESI packages from their git source trees. It handles specex separately and then iterates through other packages using setup.py.
```bash
#!/bin/bash
# This should be your actual install location...
pref="${HOME}/desi-${NERSC_HOST}"
cd specex
make clean
SPECEX_PREFIX=${pref} make -j 4 install
cd ..
for pkg in desiutil desimodel desitarget desisim specter desispec; do
cd ${pkg}
python setup.py clean
python setup.py install --prefix=${pref}
cd ..
done
```
--------------------------------
### Run desi_bias_dark_1d_model script
Source: https://desispec.readthedocs.io/en/latest/api.html
Executes the desi_bias_dark_1d_model script. No command-line arguments are specified in the example.
```bash
python3 desi_bias_dark_1d_model.py
```
--------------------------------
### Install desispec Package
Source: https://desispec.readthedocs.io/en/latest/coadd.html
Clone the desispec repository and checkout a specific development branch for co-add functionality.
```bash
git clone git@github.com:desihub/desispec.git
cd desispec
git checkout #6
```
--------------------------------
### setup_db
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/database/redshift.html
Initializes the database connection and sets up the schema and tables. It can handle both SQLite and PostgreSQL databases and supports overwriting existing data.
```APIDOC
## setup_db
### Description
Initializes the database connection and sets up the schema and tables. It can handle both SQLite and PostgreSQL databases and supports overwriting existing data.
### Parameters
- **options** (:class:`argpare.Namespace`) - Parsed command-line options.
- **kwargs** (keywords) - If present, use these instead of `options`. This is more user-friendly than setting up a :class:`~argpare.Namespace` object in, *e.g.* a Jupyter Notebook.
### Returns
- :class:`bool` - ``True`` if the configured database is a PostgreSQL database.
```
--------------------------------
### Get Spectra and Best Fit Model
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/mgii_afterburner.html
Retrieves spectra and their best-fit models from specified spectra and redrock files. Used as a starting point for MGI II afterburning.
```python
def get_spectra(spectra_name, redrock_name, lambda_width, index_to_fit,
template_dir=None, archetypes_dir=None):
"""
Get the spectra and the best fit model from a given spectra and redrock file.
Args:
spectra_name (str): The name of the spectra file.
redrock_name (str): The name of the redrock file associated to the spectra file.
```
--------------------------------
### Initialize Dark and Bias Directories
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/calibfinder.html
Sets up the paths for dark and bias calibration files. Raises an error if the dark directory is not found.
```python
self.dark_directory = f'{os.getenv("DESI_SPECTRO_DARK")}/
if not os.path.isdir(self.dark_directory):
msg = "Dark directory {} not found".format(self.dark_directory)
log.critical(msg)
raise IOError(msg)
```
--------------------------------
### Configure HARP Installation
Source: https://desispec.readthedocs.io/en/latest/install.html
Configure the HARP package for installation. Use the --prefix option to specify the installation directory. This is for installing from a released tarball.
```bash
%> cd harp-1.0.1
%> ./configure --disable-python --disable-mpi \
--prefix="${HOME}/desi-${NERSC_HOST}"
```
--------------------------------
### Install Homebrew on OS X
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Installs the Homebrew package manager on macOS systems. This is a prerequisite for installing other dependencies via Homebrew.
```bash
%> /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
```
--------------------------------
### Setup Database Connection and Schema
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/database/redshift.html
Initializes the database engine and configures schema creation and permissions. Supports both PostgreSQL and SQLite, with options for overwriting existing data.
```python
if schema:
schemaname = schema
# event.listen(Base.metadata, 'before_create', CreateSchema(schemaname))
# if overwrite:
# event.listen(Base.metadata, 'before_create',
# DDL('DROP SCHEMA IF EXISTS {0} CASCADE'.format(schemaname)))
event.listen(Base.metadata, 'before_create',
DDL(f'CREATE SCHEMA IF NOT EXISTS {schema};'))
event.listen(Base.metadata, 'after_create',
DDL(f'GRANT USAGE ON SCHEMA {schema} TO desi; GRANT SELECT ON ALL TABLES IN SCHEMA {schema} TO desi; GRANT SELECT ON ALL SEQUENCES IN SCHEMA {schema} TO desi;'))
#
# Create the file.
#
postgresql = False
if hostname:
postgresql = True
db_connection = parse_pgpass(hostname=hostname,
username=username)
if db_connection is None:
log.critical("Could not load database information!")
return 1
else:
if os.path.basename(dbfile) == dbfile:
db_file = os.path.join(options.datapath, dbfile)
else:
db_file = dbfile
if overwrite and os.path.exists(db_file):
log.info("Removing file: %s.", db_file)
os.remove(db_file)
db_connection = 'sqlite:///'+db_file
#
# SQLAlchemy stuff.
#
engine = create_engine(db_connection, echo=verbose)
dbSession.remove()
dbSession.configure(bind=engine, autoflush=False, expire_on_commit=False)
for tab in Base.metadata.tables.values():
tab.schema = schemaname
if overwrite:
log.info("Begin creating tables.")
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
log.info("Finished creating tables.")
return postgresql
```
--------------------------------
### Install Ubuntu Dependencies with apt-get
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Installs core non-Python dependencies for DESI on Ubuntu using the system package manager. Also installs pip-installable Python packages.
```bash
%> sudo apt-get install libboost-all-dev libcfitsio-dev \
libopenblas-dev liblapack-dev python3-matplotlib \
python3-scipy python3-astropy python3-requests \
python3-yaml python3-mpi4py
```
```bash
%> pip install --no-binary :all: fitsio speclite iniparser
```
--------------------------------
### Execute DESI Development Environment Setup
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Run the 'desidev' shell function to load the DESI software environment. This should be done after loading dependencies and setting up the environment.
```bash
%> desidev
```
--------------------------------
### Generate NERSC Start Time String
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/workflow/timing.html
Creates a Slurm-compatible timestamp string for the start of a night's operations. Defaults to the current night and the standard nightly start hour.
```python
def nersc_start_time(night=None, starthour=None):
"""
Transforms a night and time into a YYYY-MM-DD[THH:MM[:SS]] time string Slurm can interpret
Args:
night: str or int. In the form YYYMMDD, the night the jobs are being run.
starthour: str or int. The number of hours (between 0 and 24) after midnight where you began submitting jobs to the queue.
Returns:
str. String of the form YYYY-mm-ddTHH:MM:SS. Based on the given night and starthour
"""
if night is None:
night = what_night_is_it()
if starthour is None:
starthour = get_nightly_start_time()
starthour = int(starthour)
timetup = time.strptime(f'{night}{starthour:02d}', '%Y%m%d%H')
return nersc_format_datetime(timetup)
```
--------------------------------
### Add Header Information from Guide File
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/io/fibermap.html
Merges header keywords from a guide FITS file into the fibermap header. It checks for TILEID consistency and removes the guide file's EXPTIME if present.
```python
if guidefile is not None:
log.debug(f'Adding header keywords from {guidefile}')
guideheader = fits.getheader(guidefile, 0)
if 'TILEID' not in guideheader:
guideheader = fits.getheader(guidefile, 1)
if fibermap_header['TILEID'] != guideheader['TILEID']:
raise RuntimeError('fiberassign tile {} != guider tile {}'.format(
fibermap_header['TILEID'], guideheader['TILEID']))
if 'DEPNAM00' in guideheader:
mergedep(guideheader, fibermap_header, conflict='dst')
#- We want the spectrograph EXPTIME, not the guider EXPTIME.
#- aborted 20211127/111105 is missing guider 'EXPTIME' so check first
if 'EXPTIME' in guideheader:
guideheader.remove('EXPTIME')
fibermap_header.extend(guideheader, strip=True, unique=True)
```
--------------------------------
### Header Dependency Setup
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/preproc.html
This snippet demonstrates how to set up header dependencies for calibration data, including environment variables like `DESI_SPECTRO_CALIB`, `DESI_SPECTRO_REDUX`, and `SPECPROD`.
```python
header = header.copy()
depend.setdep(header, 'DESI_SPECTRO_CALIB', os.getenv('DESI_SPECTRO_CALIB'))
for key in ['DESI_SPECTRO_REDUX', 'SPECPROD']:
if key in os.environ:
depend.setdep(header, key, os.environ[key])
```
--------------------------------
### Install Remaining Python Dependencies with pip on OS X
Source: https://desispec.readthedocs.io/en/latest/install.html
Installs the rest of the required Python packages using pip3.
```bash
%> pip3 install requests pyyaml iniparser speclite astropy
%> pip3 install --no-binary :all: fitsio
```
--------------------------------
### Timer Initialization and Setup
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/scripts/zproc.html
Initializes a Timer object, controlling log message output based on MPI rank. Sets up timing for various startup phases, including imports and MPI connection.
```python
error_count = 0
if rank == 0:
thisfile=os.path.dirname(os.path.abspath(__file__))
thistime=datetime.datetime.fromtimestamp(start_imports).isoformat()
log.info(f'rank 0 started {thisfile} at {thistime}')
## Start timer; only print log messages from rank 0 (others are silent)
timer = desiutil.timer.Timer(silent=rank>0)
## Fill in timing information for steps before we had the timer created
if args.starttime is not None:
timer.start('startup', starttime=args.starttime)
timer.stop('startup', stoptime=start_imports)
timer.start('imports', starttime=start_imports)
timer.stop('imports', stoptime=stop_imports)
timer.start('mpi_connect', starttime=start_mpi_connect)
timer.stop('mpi_connect', stoptime=stop_mpi_connect)
```
--------------------------------
### Initialize Wavelength, Flux, and Ivar Arrays
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/io/emlinefit.html
Initializes empty arrays for wavelength, flux, and inverse variance, shaped to accommodate data from multiple cameras.
```python
# AR b : 3600,5800 , r : 5760,7620 , z : 7520,9824
# AR we keep all data points (i.e. including overlaps)
# AR note that waves is not ordered, as a consequence
waves = np.zeros(0)
fluxes = np.zeros(0).reshape(nspec, 0)
ivars = np.zeros(0).reshape(nspec, 0)
```
--------------------------------
### Install DESI Dependencies on Ubuntu
Source: https://desispec.readthedocs.io/en/latest/install.html
Installs essential non-Python and Python dependencies for the DESI pipeline on Ubuntu using apt-get and pip.
```bash
%> sudo apt-get install libboost-all-dev libcfitsio-dev \
libopenblas-dev liblapack-dev python3-matplotlib \
python3-scipy python3-astropy python3-requests \
python3-yaml python3-mpi4py
%> pip install --no-binary :all: fitsio speclite iniparser
```
--------------------------------
### desispec.database.redshift.setup_db
Source: https://desispec.readthedocs.io/en/latest/api.html
Initializes the database connection for Redshift. It can accept parsed command-line options or keyword arguments.
```APIDOC
## desispec.database.redshift.setup_db
### Description
Initialize the database connection.
### Parameters
#### Path Parameters
- **options** (argpare.Namespace) - Optional - Parsed command-line options.
- **kwargs** (keywords) - Optional - If present, use these instead of options. This is more user-friendly than setting up a `Namespace` object in, _e.g._ a Jupyter Notebook.
### Returns
- `True` if the configured database is a PostgreSQL database.
```
--------------------------------
### Main Function for Tile QA Script
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/scripts/tileqa.html
Initializes logging, parses command-line arguments, sets default production and output directories if not provided, and logs the production path being used.
```python
def main(args=None):
log = get_logger()
if not isinstance(args, argparse.Namespace):
args = parse(options=args)
if args.prod is None:
args.prod = specprod_root()
elif args.prod.find("/")<0 :
args.prod = specprod_root(args.prod)
if args.outdir is None :
args.outdir = args.prod
if args.exposure_qa_dir is None :
args.exposure_qa_dir = args.prod
log.info('prod = {}'.format(args.prod))
```
--------------------------------
### Install Python Dependencies with Homebrew on OS X
Source: https://desispec.readthedocs.io/en/latest/install.html
Installs Python-specific dependencies using Homebrew, ensuring Python 3 versions are used.
```bash
%> brew install homebrew/python/mpi4py --with-python3
%> brew install homebrew/python/scipy --with-python3
%> brew install homebrew/python/matplotlib --with-python3
```
--------------------------------
### Main Entry Point for Database Loading
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/database/redshift.html
The main function orchestrates the database loading process. It parses arguments, sets up logging, initializes the database connection, and loads configuration for various data types.
```python
def main():
"""Entry point for command-line script.
Returns
-------
:class:`int`
An integer suitable for passing to :func:`sys.exit`.
"""
global log
freeze_iers()
#
# command-line arguments
#
options = get_options()
#
# Logging
#
if options.verbose:
log = get_logger(DEBUG, timestamp=True)
else:
log = get_logger(INFO, timestamp=True)
#
# Initialize DB
#
postgresql = setup_db(options)
#
# Load configuration
#
loaders = {'exposures': [{'filepaths': os.path.join(options.datapath, 'spectro', 'redux', os.environ['SPECPROD'], 'tiles-{specprod}.fits'.format(specprod=os.environ['SPECPROD'])),
'tcls': Tile,
'hdu': 'TILE_COMPLETENESS',
'q3c': 'tilera',
'chunksize': options.chunksize,
'maxrows': options.maxrows
},
{'filepaths': os.path.join(options.datapath, 'spectro', 'redux', os.environ['SPECPROD'], 'exposures-{specprod}.fits'.format(specprod=os.environ['SPECPROD'])),
'tcls': Exposure,
'hdu': 'EXPOSURES',
'insert': {'mjd': ('date_obs',)},
'convert': {'date_obs': lambda x: Time(x, format='mjd').to_value('datetime').replace(tzinfo=utc)},
'q3c': 'tilera',
'chunksize': options.chunksize,
'maxrows': options.maxrows
},
{'filepaths': os.path.join(options.datapath, 'spectro', 'redux', os.environ['SPECPROD'], 'exposures-{specprod}.fits'.format(specprod=os.environ['SPECPROD'])),
'tcls': Frame,
'hdu': 'FRAMES',
'preload': _frameid,
'chunksize': options.chunksize,
'maxrows': options.maxrows
}],
#
# The potential targets are supposed to include data for all targets.
# In other words, every actual target is also a potential target.
#
'photometry': [{'filepaths': glob.glob(os.path.join(options.targetpath, 'vac', 'lsdr9-photometry', os.environ['SPECPROD'], 'v1.0', 'potential-targets', 'tractorphot', 'tractorphot*.fits')),
'tcls': Photometry,
'hdu': 'TRACTORPHOT',
'expand': {'DCHISQ': ('dchisq_psf', 'dchisq_rex', 'dchisq_dev', 'dchisq_exp', 'dchisq_ser',),
'OBJID': 'brick_objid',
'TYPE': 'morphtype'},
# 'rowfilter': _remove_loaded_targetid,
'chunksize': options.chunksize,
```
--------------------------------
### desispec.bootcalib.script_bootcalib
Source: https://desispec.readthedocs.io/en/latest/api.html
Runs desi_bootcalib on a series of preproc files.
```APIDOC
## desispec.bootcalib.script_bootcalib
### Description
Runs desi_bootcalib on a series of preproc files.
### Parameters
* **arc_idx** (list) – List of arc indices.
* **flat_idx** (list) – List of flat indices.
* **cameras** (list, optional) – List of cameras to process. Defaults to None.
* **channels** (list, optional) – List of channels to process. Defaults to None.
* **nproc** (int, optional) – Number of processes to use. Defaults to 10.
### Returns
* A list of processed files.
```
--------------------------------
### Create Python Package Directory
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Pre-create the Python package directory structure required for installing packages. This ensures the directory exists before installation begins.
```bash
%> mkdir -p ${HOME}/desi-${NERSC_HOST}/lib/python3.5/site-packages
```
--------------------------------
### Parse Command-Line Arguments
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/scripts/qsoqn.html
Sets up an argument parser for the qsoqn script, defining options for input files, output paths, target selection, and parameters for both QuasarNet (QN) and Redrock (RR) runs.
```python
import argparse
def parse(options=None):
parser = argparse.ArgumentParser(description="Run QN and rerun RR (only for SPECTYPE != QSO or for SPECTYPE == QSO && |z_QN - z_RR| > 0.05) on a coadd file to find true quasars with correct redshift")
parser.add_argument("--coadd", type=str, required=True,
help="coadd file containing spectra")
parser.add_argument("--redrock", type=str, required=True,
help="redrock file associated (in the same folder) to the coadd file")
parser.add_argument("--output", type=str, required=True,
help="output filename where the result of the QN will be saved")
parser.add_argument("--target_selection", type=str, required=False, default="qso_targets",
help="on which sample the QN is performed:
qso_targets (QSO targets) -- qso (works also)
/ all_targets (All targets in the coadd file) -- all (works also)")
parser.add_argument("--save_target", type=str, required=False, default="restricted",
help="which objects will be saved in the output files:
restricted (objects which are identify as QSO by QN and where we have a new run of RR)
/ all (All objects which are tested by QN
--> To have coadd.size objects in the ouput file: set --target_selection all_targets --save_target all")
# for QN
parser.add_argument("--c_thresh", type=float, required=False, default=0.5,
help="For QN: is_qso_QN = np.sum(c_line > c_thresh, axis=0) >= n_thresh")
parser.add_argument("--n_thresh", type=int, required=False, default=1,
help="For QN: is_qso_QN = np.sum(c_line > c_thresh, axis=0) >= n_thresh")
# for RR
parser.add_argument("--templates", type=str, nargs='+', required=False,
help="give the templates used during the new run of RR
By default use the templates from redrock of the form rrtemplate-QSO-*.fits")
parser.add_argument("--filename_priors", type=str, required=False, default=None,
help="filename for the RR prior file, by default use the directory of coadd file")
parser.add_argument("--filename_output_rerun_RR", type=str, required=False, default=None,
help="filename for the output of the new run of RR, by default use the directory of coadd file")
parser.add_argument("--filename_redrock_rerun_RR", type=str, required=False, default=None,
help="filename for the redrock file of the new run of RR, by default use the directory of coadd file")
parser.add_argument("--delete_RR_output", type=str, required=False, default='True',
help="delete RR outputs: True or False, they are useless since everything usefull are saved in output, by defaut:True")
if options is None:
args = parser.parse_args()
else:
args = parser.parse_args(options)
if args.templates is None:
args.templates = get_default_qso_templates()
return args
```
--------------------------------
### Set up DESI Environment
Source: https://desispec.readthedocs.io/en/latest/coadd.html
Source the DESI environment script to set up necessary variables for using DESI software on Cori.
```bash
source /project/projectdirs/desi/software/desi_environment.sh
```
--------------------------------
### Install OS X Dependencies with Homebrew
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Installs CFITSIO, BOOST, and OpenMPI using Homebrew on macOS. These are essential non-Python dependencies for the DESI pipeline.
```bash
%> brew install cfitsio
```
```bash
%> brew install boost
```
```bash
%> brew install openmpi
```
--------------------------------
### Initialize Photometric Systems and Filters
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/scripts/stdstars.html
Initializes photometric systems and loads filter curves for Legacy Survey and Gaia. This is used to prepare for magnitude calculations.
```python
log.warning(" EBV = 0.0")
fibermap['PHOTSYS'] = 'S'
fibermap['EBV'] = 0.0
if not np.isin(np.unique(fibermap['PHOTSYS']),['','N','S','G']).all():
log.error('Unknown PHOTSYS found')
raise Exception('Unknown PHOTSYS found')
# Fetching Filter curves
model_filters = dict()
for band in ["G","R","Z"] :
for photsys in np.unique(fibermap['PHOTSYS']) :
if photsys in ['N','S']:
model_filters[band+photsys] = load_legacy_survey_filter(band=band,photsys=photsys)
if len(model_filters) == 0:
log.info('No Legacy survey photometry identified in fibermap')
# I will always load gaia data even if we are fitting LS standards only
for band in ["G", "BP", "RP"] :
model_filters["GAIA-" + band] = load_gaia_filter(band=band, dr=2)
```
--------------------------------
### Install Python Stack on OS X with Homebrew
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Installs Python 3 versions of mpi4py, scipy, and matplotlib using Homebrew on macOS. This sets up the Python environment for DESI.
```bash
%> brew install homebrew/python/mpi4py --with-python3
```
```bash
%> brew install homebrew/python/scipy --with-python3
```
```bash
%> brew install homebrew/python/matplotlib --with-python3
```
--------------------------------
### Initialize XYTraceSet
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/xytraceset.html
Initializes the XYTraceSet wrapper with Legendre coefficients for x and y traces, wavelength range, and pixel dimensions. Optional coefficients for x and y signal and metadata can also be provided.
```python
from desiutil.log import get_logger
import numpy as np
class XYTraceSet(object):
def __init__(self, xcoef, ycoef, wavemin, wavemax, npix_y, xsigcoef = None, ysigcoef = None, meta = None) :
"""
Lightweight wrapper for trace coordinates and wavelength solution
Args:
xcoef: 2D[ntrace, ncoef] Legendre coefficient of x as a function of wavelength
ycoef: 2D[ntrace, ncoef] Legendre coefficient of y as a function of wavelength
wavemin : float
wavemax : float. wavemin and wavemax are used to define a reduced variable legx(wave,wavemin,wavemax)=2*(wave-wavemin)/(wavemax-wavemin)-1
used to compute the traces, xccd=legval(legx(wave,wavemin,wavemax),xtrace[fiber])
"""
from specter.util.traceset import TraceSet
assert(xcoef.shape[0] == ycoef.shape[0])
if xsigcoef is not None :
assert(xcoef.shape[0] == xsigcoef.shape[0])
if ysigcoef is not None :
assert(xcoef.shape[0] == ysigcoef.shape[0])
self.nspec = xcoef.shape[0]
self.wavemin = wavemin
self.wavemax = wavemax
self.npix_y = npix_y
self._xcoef = xcoef
self._ycoef = ycoef
self._xsigcoef = xsigcoef
self._ysigcoef = ysigcoef
self.x_vs_wave_traceset = TraceSet(xcoef,[wavemin,wavemax])
self.y_vs_wave_traceset = TraceSet(ycoef,[wavemin,wavemax])
self.xsig_vs_wave_traceset = None
self.ysig_vs_wave_traceset = None
if xsigcoef is not None :
self.xsig_vs_wave_traceset = TraceSet(xsigcoef,[wavemin,wavemax])
if ysigcoef is not None :
self.ysig_vs_wave_traceset = TraceSet(ysigcoef,[wavemin,wavemax])
self.wave_vs_y_traceset = None
self.meta = meta
```
--------------------------------
### Install Remaining OS X Dependencies with pip3
Source: https://desispec.readthedocs.io/en/latest/_sources/install.rst.txt
Installs Python packages like requests, pyyaml, speclite, astropy, and fitsio using pip3 on macOS. Ensures all necessary Python libraries are available.
```bash
%> pip3 install requests pyyaml iniparser speclite astropy
```
```bash
%> pip3 install --no-binary :all: fitsio
```
--------------------------------
### Get Fiber Bitmask Comparison Value
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/fiberbitmasking.html
Translates a string designation for a reduction step (e.g., 'fluxcalib', 'sky') and camera band ('b', 'r', 'z') into a 32-bit integer bitmask. Use this to get the appropriate bitmask for specific operations.
```python
def get_fiberbitmask_comparison_value(kind,band):
"""Takes a string argument and returns a 32-bit integer representing the logical OR of all
fatally bad fibermask bits for that given reduction step
Args:
kind: str : string designating which combination of bits to use based on the operation.
Possible values are "all", "sky" (or "skysub"), "flat",
"flux" (or "fluxcalib"), "star" (or "stdstars")
band: str : BADAMP band bits to set. Values include 'b', 'r', 'z', or
combinations thereof such as 'brz'
Returns:
bitmask : 32 bit bitmask corresponding to the fiberbitmask of the desired kind
in the desired cameras (bands).
if FIBERSTATUS & bitmask != 0, then that fiber should not be used
"""
if kind.lower() == 'all':
return get_all_fiberbitmask_with_amp(band)
elif kind.lower()[:3] == 'sky':
return get_skysub_fiberbitmask_val(band)
elif kind.lower() == 'flat':
return get_flat_fiberbitmask_val(band)
elif 'star' in kind.lower():
return get_stdstars_fiberbitmask_val(band)
elif 'flux' in kind.lower():
return get_fluxcalib_fiberbitmask_val(band)
else:
log = get_logger()
```
--------------------------------
### Initialize DESI Dashboard Class
Source: https://desispec.readthedocs.io/en/latest/_modules/desispec/desi_dashboard.html
Initializes the DESI_DASHBOARD class by parsing command-line arguments for product name, directories, and URLs. Sets up database connections and computes schema information.
```python
import argparse
import os
import fitsio
import astropy.io.fits as pyfits
import subprocess
import pandas as pd
import time
import numpy as np
import psycopg2
import hashlib
import pdb
[docs]
class DESI_DASHBOARD(object):
""" Code to generate the statistic of desi_pipe production status
Usage:
python3 desi_dashboard.py --prod realtime10 --prod_dir /global/cscratch1/sd/zhangkai/desi/ --output_dir /global/project/projectdirs/desi/www/users/zhangkai/desi_dashboard/ --output_url https://portal.nersc.gov/project/desi/users/zhangkai/desi_dashboard/
"""
def __init__(self):
############
## Input ###
############
parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser = self._init_parser(parser)
args = parser.parse_args()
prod=args.prod #product name
prod_dir=args.prod_dir # base directory of product
self.output_dir=args.output_dir+prod+"/" # Portal directory for output html files
self.output_url=args.output_url+prod+"/" # corresponding URL
######################################################################
## sub directories. Should not change if generated by the same code ##
## that follows the same directory strucure ##
######################################################################
self.data_dir=prod_dir+prod+"/spectro/data/"
self.log_dir=prod_dir+prod+"/spectro/redux/daily/run/logs"
self.redux_dir=prod_dir+prod+"/spectro/redux/daily"
#############
## Setup ####
#############
self.conn=self.get_db_conn(host="nerscdb03.nersc.gov",database="desidev",user="desidev_admin")
self.cur=self.conn.cursor()
self.schema=self._compute_schema(self.redux_dir)
self.tasktype_arr=['preproc','psf','psfnight','traceshift','extract','fiberflat','fiberflatnight','sky','starfit','fluxcalib','cframe','spectra','redshift']
self.tasktype_arr_nonight=['spectra','redshift']
############
# Load data
############
self.file_count=self.count_files()
for tasktype in self.tasktype_arr:
cmd="self.get_table(tasktype='"+tasktype+"')"
exec(cmd)
nights=np.unique(self.df_preproc['night'])
strTable=self._initialize_page()
strTable=strTable+""
strTable=strTable+"
Data Dir: "+self.redux_dir+"
"
#########################
#### Overall Table ######
#########################
table=self._compute_night_statistic("all")
strTable=strTable+self._add_html_table_with_link(table,"Overall")
####################################
#### Table for individual night ####
####################################
for night in nights:
# Create Statistic table for each night
table=self._compute_night_statistic(night)
strTable=strTable+self._add_html_table(table,str(night))
timestamp=time.strftime("%Y-%m-%d %H:%M:%S",time.localtime())
print(timestamp)
strTable=strTable+"