### Install CFITSIO, Boost, and OpenMPI with Homebrew Source: https://github.com/desihub/desispec/blob/main/doc/install.md Installs core non-Python dependencies using Homebrew on macOS. ```bash ட்டுகள்%> brew install cfitsio ட்டுகள்%> brew install boost ட்டுகள்%> brew install openmpi ``` -------------------------------- ### Execute DESI Software Installation Script Source: https://github.com/desihub/desispec/blob/main/doc/install.md Run the 'install.sh' script to install DESI software from source. Ensure the script is executable before running. ```bash ./install.sh ``` -------------------------------- ### Create DESI Software Installation Script Source: https://github.com/desihub/desispec/blob/main/doc/install.md Create a bash script to automate the installation of DESI packages from their git source trees. This script cleans and installs each package. ```bash %> cd $HOME/git-$NERSC_HOST %> cat install.sh #!/bin/bash # This should be your actual install location... pref="${HOME}/desi-${NERSC_HOST}" cd specex make clean SPECEX_PREFIX=${pref} make -j 4 install cd .. for pkg in desiutil desimodel desitarget desisim specter desispec; do cd ${pkg} python setup.py clean python setup.py install --prefix=${pref} cd .. done ``` -------------------------------- ### Install Symlink for Development Source: https://github.com/desihub/desispec/blob/main/README.rst Use this command to install a symlink to your live git checkout for development purposes. Use the --uninstall flag to remove it. ```bash python setup.py develop --prefix=/path/to/somewhere ``` ```bash python setup.py develop --prefix=/path/to/somewhere --uninstall ``` -------------------------------- ### Displaying Coordinate and Guide File Paths Source: https://github.com/desihub/desispec/blob/main/doc/nb/FibermapCombinationTest.ipynb Shows the determined paths for the coordinates and guide files. ```python coordfile, guidefile ``` -------------------------------- ### Install Fixed Version Source: https://github.com/desihub/desispec/blob/main/README.rst Installs a fixed version of the tools to a specified prefix. This is useful for creating stable installations. ```bash python setup.py install --prefix=/path/to/somewhere ``` -------------------------------- ### Execute DESI Environment Setup Source: https://github.com/desihub/desispec/blob/main/doc/install.md Run the 'desidev' shell function to load the DESI software environment. ```bash %> desidev ``` -------------------------------- ### Install DESI Dependencies on Ubuntu Source: https://github.com/desihub/desispec/blob/main/doc/install.md Installs essential non-Python and Python dependencies for DESI on Ubuntu using apt-get and pip. Use --no-binary :all: for fitsio and speclite to ensure they are built from source. ```bash ட்டுகள்%> sudo apt-get install libboost-all-dev libcfitsio-dev \ libopenblas-dev liblapack-dev python3-matplotlib \ python3-scipy python3-astropy python3-requests \ python3-yaml python3-mpi4py ட்டுகள்%> pip install --no-binary :all: fitsio speclite iniparser ``` -------------------------------- ### Install Remaining Python Dependencies on OS X Source: https://github.com/desihub/desispec/blob/main/doc/install.md Installs the rest of the required Python packages using pip3, with a specific flag for fitsio to ensure it's built from source. ```bash ட்டுகள்%> pip3 install requests pyyaml iniparser speclite astropy ட்டுகள்%> pip3 install --no-binary :all: fitsio ``` -------------------------------- ### Record Input Guide and Coordinates Files Source: https://github.com/desihub/desispec/blob/main/doc/nb/FibermapCombinationTest.ipynb Records the base filenames of the input guide and coordinate files into the fibermap header. If a file is not provided, 'MISSING' is recorded. ```python if guidefile is not None: fibermap_header['GUIDEFIL'] = os.path.basename(guidefile) else: fibermap_header['GUIDEFIL'] = 'MISSING' if coordfile is not None: fibermap_header['COORDFIL'] = os.path.basename(coordfile) else: fibermap_header['COORDFIL'] = 'MISSING' ``` -------------------------------- ### Configure HARP Release Tarball Installation Source: https://github.com/desihub/desispec/blob/main/doc/install.md Configure the HARP package for installation from a release tarball. Use '--prefix' to specify the installation directory. ```bash %> cd harp-1.0.1 %> ./configure --disable-python --disable-mpi \ --prefix="${HOME}/desi-${NERSC_HOST}" ``` -------------------------------- ### Finding Coordinates and Guide Files Source: https://github.com/desihub/desispec/blob/main/doc/nb/FibermapCombinationTest.ipynb Locates the coordinates and guide files based on the fiber assignment file directory. Includes logic to handle cases with zero, one, or multiple matching files, and options to force continuation or raise errors. ```python force=args.force #- Find coordinates file in same directory dirname, filename = os.path.split(rawfafile) globfiles = glob.glob(dirname+'/coordinates-*.fits') if len(globfiles) == 1: coordfile = globfiles[0] elif len(globfiles) == 0: message = f'No coordinates*.fits file in fiberassign dir {dirname}' if force: log.error(message + '; continuing anyway') coordfile = None else: raise FileNotFoundError(message) elif len(globfiles) > 1: raise RuntimeError(f'Multiple coordinates*.fits files in fiberassign dir {dirname}') #- And guide file dirname, filename = os.path.split(rawfafile) globfiles = glob.glob(dirname+'/guide-????????.fits.fz') if len(globfiles) == 0: #- try falling back to acquisition image globfiles = glob.glob(dirname+'/guide-????????-0000.fits.fz') if len(globfiles) == 1: guidefile = globfiles[0] elif len(globfiles) == 0: message = f'No guide-*.fits.fz file in fiberassign dir {dirname}' if force: log.error(message + '; continuing anyway') guidefile = None else: raise FileNotFoundError(message) elif len(globfiles) > 1: raise RuntimeError(f'Multiple guide-*.fits.fz files in fiberassign dir {dirname}') ``` -------------------------------- ### Edison Environment Setup and Sky Computation Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_Frame.ipynb Commands to set up the environment on Edison and run the desi_compute_sky.py script with specified input and output files. ```bash [edison ~] source env.sh [edison ~] cd Python/desispec [edison ~] git pull [edison ~] cd [edison ~] module use ~xavier/modules [edison ~] module switch desispec/qa_exposure [edison ~] cd qa_test desi_compute_sky.py \ --infile /project/projectdirs/desi/spectro/redux/sjb/dogwood/exposures/20150211/00000002/frame-b0-00000002.fits \ --fibermap /project/projectdirs/desi/spectro/sim/alpha-5/20150211/fibermap-00000002.fits \ --fiberflat /project/projectdirs/desi/spectro/redux/sjb/dogwood/calib2d/20150211/fiberflat-b0-00000001.fits \ --outfile sky-b0-00000002.fits \ --qafile qa-b0-00000002.yaml \ --qafig qa-b0-00000002.pdf \ > qa_tst.log ``` -------------------------------- ### Process DESI Guide Star FITS Files Source: https://github.com/desihub/desispec/blob/main/doc/nb/cosmic-joiner.ipynb Iterates through guide star FITS files for a given date, extracting and printing TILEID, TARGTRA, and TARGTDEC from the headers. Requires the 'glob', 'os', and 'fits' libraries. ```python import glob import os from astropy.io import fits def guides(date): names = glob.glob(os.path.join("/exposures","desi", date, "*", "guide-0*.fits.fz")) for n in names: with fits.open(n) as hdus: print(n) print("TILE", hdus[0].header["TILEID"]) print("RA", hdus[0].header["TARGTRA"]) print("DEC", hdus[0].header["TARGTDEC"]) print() ``` -------------------------------- ### Example Call to `desi_preproc` Source: https://github.com/desihub/desispec/blob/main/doc/nb/PreprocConstructionTest.ipynb Demonstrates how to call the `desi_preproc` script with various arguments, including night, exposure ID, camera, fibermap, and output file. This is useful for understanding the script's basic usage. ```bash desi_preproc -n 20210412 -e 84523 --cameras r0 \ --fibermap $DESI_ROOT/spectro/redux/f3/preproc/20210412/00084523/fibermap-00084523.fits \ --outfile preproc-r0-00084523.fits --model-variance ``` -------------------------------- ### Setup database connection for target schema Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Establishes a database connection to a specific schema ('specprod_target') on a given host, with options for overwriting and verbosity. ```python postgresql = dsr.setup_db(schema=specprod+'_target', overwrite=overwrite, hostname='nerscdb03.nersc.gov', verbose=True) ``` -------------------------------- ### Show Image with Ginga (Commented Out) Source: https://github.com/desihub/desispec/blob/main/doc/nb/Preproc.ipynb A commented-out example showing how to display image data using the pypeit.ginga viewer. This is typically used for interactive inspection of astronomical images. ```python #ginga.show_image(z2_hdu.data) ``` -------------------------------- ### Install Homebrew on OS X Source: https://github.com/desihub/desispec/blob/main/doc/install.md Installs Homebrew, a package manager for macOS, which is used to install other dependencies. ```bash ட்டுகள்%> /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" ``` -------------------------------- ### srun Command for `desi_preproc` Source: https://github.com/desihub/desispec/blob/main/doc/nb/PreprocConstructionTest.ipynb Shows an example of how to run `desi_preproc` using `srun` for parallel processing. This command includes options for node and task count, CPU binding, and various pre-processing flags. ```bash srun -N 3 -n 200 -c 4 --cpu-bind=cores desi_proc --traceshift --nostdstarfit --nofluxcalib --cameras a0123456789 -n 20210412 -e 84523 --timingfile /global/cfs/cdirs/desi/spectro/redux/f3/run/scripts/night/20210412/prestdstar-20210412-00084523-a0123456789-timing-$SLURM_JOBID.json --mpi --nofluxcalib ``` -------------------------------- ### Install DESI Coadd Development Branch Source: https://github.com/desihub/desispec/blob/main/doc/coadd.md Clone the desispec repository and checkout the co-add development branch for testing. ```bash git clone git@github.com:desihub/desispec.git cd desispec git checkout #6 ``` -------------------------------- ### Create Python Package Directory Source: https://github.com/desihub/desispec/blob/main/doc/install.md Pre-create the necessary directory for Python packages before installing software. ```bash %> mkdir -p ${HOME}/desi-${NERSC_HOST}/lib/python3.5/site-packages ``` -------------------------------- ### Spectra Object Construction and Basic Usage Source: https://context7.com/desihub/desispec/llms.txt Demonstrates how to construct a Spectra object manually and perform basic operations like getting the number of spectra and targets. ```APIDOC ## Spectra Object Construction and Basic Usage ### Description This section shows how to create a `Spectra` object from scratch using provided data for different bands, and then how to query basic properties like the number of spectra and targets. ### Method `Spectra(bands, wave, flux, ivar, mask, fibermap, ...)` ### Parameters - **bands** (list of str) - List of band names (e.g., ['b', 'r', 'z']). - **wave** (dict) - Dictionary where keys are band names and values are numpy arrays of wavelengths. - **flux** (dict) - Dictionary where keys are band names and values are numpy arrays of flux. - **ivar** (dict) - Dictionary where keys are band names and values are numpy arrays of inverse variance. - **mask** (dict) - Dictionary where keys are band names and values are numpy arrays of masks. - **fibermap** (Table) - Astropy Table containing fiber information. ### Request Example ```python import numpy as np from desispec.spectra import Spectra from astropy.table import Table bands = ['b', 'r', 'z'] nspec = 5 wave = { 'b': np.linspace(3600., 5800., 1000), 'r': np.linspace(5800., 7600., 1000), 'z': np.linspace(7600., 9800., 1000), } flux = {b: np.random.normal(0, 1, (nspec, 1000)).astype(np.float32) for b in bands} ivar = {b: np.ones((nspec, 1000), dtype=np.float32) for b in bands} mask = {b: np.zeros((nspec, 1000), dtype=np.uint32) for b in bands} fmap = Table({'TARGETID': np.arange(nspec, dtype=np.int64), 'TARGET_RA': np.zeros(nspec), 'TARGET_DEC': np.zeros(nspec)}) sp = Spectra(bands=bands, wave=wave, flux=flux, ivar=ivar, mask=mask, fibermap=fmap) print(sp.num_spectra()) # 5 print(sp.num_targets()) # 5 ``` ### Response #### Success Response (200) - **num_spectra()** (int) - Returns the total number of spectra. - **num_targets()** (int) - Returns the total number of unique targets. #### Response Example ``` 5 5 ``` ``` -------------------------------- ### Display Image with Ginga Source: https://github.com/desihub/desispec/blob/main/doc/nb/Overscan.ipynb Displays an image using the Ginga toolkit. Ensure Ginga is installed and configured. ```python ginga.show_image(img_z3.pix) ``` -------------------------------- ### Add Header Info from Guide File Source: https://github.com/desihub/desispec/blob/main/doc/nb/FibermapCombinationTest.ipynb Adds header keywords from a guide file to the fibermap. It handles cases where the full header might be in HDU 0 or HDU 1 and raises an error if TILEIDs do not match. ```python if guidefile is not None: log.debug(f'Adding header keywords from {guidefile}') guideheader = fits.getheader(guidefile, 0) if 'TILEID' not in guideheader: guideheader = fits.getheader(guidefile, 1) if fibermap_header['TILEID'] != guideheader['TILEID']: raise RuntimeError('fiberassign tile {} != guider tile {}'.format( fibermap_header['TILEID'], guideheader['TILEID'])) # addkeys(fibermap.meta, guideheader, skipkeys=skipkeys) fibermap_header.extend(guideheader, strip=True) fibermap_header.remove('EXPTIME') ``` -------------------------------- ### Define potential target photometry file path Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Defines the file path for potential target photometry data. This is a commented-out example. ```python # targetphot_potential_file = os.path.join(os.environ['DESI_ROOT'], 'public', release, 'vac', 'lsdr9-photometry', os.environ['SPECPROD'], targetphot_version, 'potential-targets', f"targetphot-potential-{os.environ['SPECPROD']}.fits") ``` -------------------------------- ### Import DESI Libraries Source: https://github.com/desihub/desispec/blob/main/doc/nb/Intro_to_DESI_spectra.ipynb Imports essential libraries for DESI spectral analysis, including numpy, healpy, fitsio, and matplotlib. Ensure these are installed before running. ```python import os import numpy as np import healpy as hp from glob import glob import fitsio from collections import defaultdict from desitarget import desi_mask import matplotlib.pyplot as plt %pylab inline ``` -------------------------------- ### Get HEALPix Pixel Area using healpy Source: https://github.com/desihub/desispec/blob/main/doc/nb/Intro_to_DESI_spectra.ipynb This snippet demonstrates a more direct method to obtain the area of a HEALPix pixel for a given nside using the healpy library. It requires the healpy library to be installed. ```python hp.nside2pixarea(64,degrees=True) ``` -------------------------------- ### Set Up Preprocessing Variables Source: https://github.com/desihub/desispec/blob/main/doc/nb/PreprocConstructionTest.ipynb Configures environment variables and defines input/output file paths for the preprocessing step. This includes setting the log level, night, exposure ID, camera, and constructing the full paths for input raw data, fibermap, and output preproc file. ```python os.environ['DESI_LOGLEVEL'] = 'DEBUG' night = 20210412 expid = 84523 camera = 'r0' infile = findfile('raw', night, expid) fibermap = os.path.join(os.environ['DESI_SPECTRO_REDUX'], os.environ['SPECPROD'], 'preproc', str(night), f'{expid:08d}', f'fibermap-{expid:08d}.fits') outfile = os.path.join(os.environ['CSCRATCH'], f'preproc-{camera}-{expid:08d}.fits') infile, fibermap, outfile ``` -------------------------------- ### Run desi_bootcalib for z0 (Wavelengths) Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_fulltest.ipynb Execute the `desi_bootcalib.py` script for the z0 band using files from the Wavelengths directory, specifying fiberflat, arcfile, output, and QA files. Use the `--test` flag for testing. ```bash desi_bootcalib.py \ --fiberflat /Users/xavier/DESI/Wavelengths/pix-z0-00000001.fits \ --arcfile /Users/xavier/DESI/Wavelengths/pix-z0-00000000.fits \ --outfile /Users/xavier/DESI/Wavelengths/boot_psf-z0.fits \ --qafile /Users/xavier/DESI/Wavelengths/qa_boot-z0.pdf \ --test ``` -------------------------------- ### Run script_bootcalib with Python Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_fulltest.ipynb Import and execute the `script_bootcalib` function from `desispec.bootcalib` in Python, passing lists of integers as arguments. ```python from desispec.bootcalib import script_bootcalib as scb scb([0,1,2,3,4,5,6,7,8,9], [10,11,12,13,14]) ``` -------------------------------- ### Set up DESI Environment Source: https://github.com/desihub/desispec/blob/main/doc/coadd.md Source the DESI environment script to set up necessary paths and configurations for using DESI software. ```bash source /project/projectdirs/desi/software/desi_environment.sh ``` -------------------------------- ### Define file loading configuration Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Sets up a list of dictionaries, where each dictionary specifies parameters for loading different targetphot FITS files into the database, including file paths, table class, HДУ, expansion rules, and chunking. ```python loader = [{'filepaths': [os.path.join('/global/cscratch1/sd/ioannis/photocatalog', os.environ['SPECPROD'], 'targetphot-{specprod}.fits'.format(specprod=os.environ['SPECPROD'])), os.path.join('/global/cscratch1/sd/ioannis/photocatalog', os.environ['SPECPROD'], 'targetphot-potential-targets-{specprod}.fits'.format(specprod=os.environ['SPECPROD'])), os.path.join('/global/cscratch1/sd/ioannis/photocatalog', os.environ['SPECPROD'], 'targetphot-missing-{specprod}.fits'.format(specprod=os.environ['SPECPROD']))], 'tcls': dsr.Target, 'hdu': 'TARGETPHOT', 'expand': {'DCHISQ': ('dchisq_psf', 'dchisq_rex', 'dchisq_dev', 'dchisq_exp', 'dchisq_ser',)}, 'q3c': 'ra', 'chunksize': 100000, 'maxrows': 0 },] ``` -------------------------------- ### Install Python Dependencies with Homebrew on OS X Source: https://github.com/desihub/desispec/blob/main/doc/install.md Installs Python 3 versions of mpi4py and scipy using Homebrew on macOS. The --with-python3 flag ensures compatibility. ```bash ட்டுகள்%> brew install homebrew/python/mpi4py --with-python3 ட்டுகள்%> brew install homebrew/python/scipy --with-python3 ட்டுகள்%> brew install homebrew/python/matplotlib --with-python3 ``` -------------------------------- ### Run desi_bootcalib for r0 Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_fulltest.ipynb Execute the `desi_bootcalib.py` script for the r0 band, specifying fiberflat, arcfile, output, and QA files. Use the `--test` flag for testing. ```bash desi_bootcalib.py \ --fiberflat /Users/xavier/DESI/Wavelengths/pix-r0-00000001.fits \ --arcfile /Users/xavier/DESI/Wavelengths/pix-r0-00000000.fits \ --outfile /Users/xavier/DESI/Wavelengths/boot_psf-r0.fits \ --qafile /Users/xavier/DESI/Wavelengths/qa_boot-r0.pdf \ --test ``` -------------------------------- ### Run desi_bootcalib for z0 (NERSC_TED) Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_fulltest.ipynb Execute the `desi_bootcalib.py` script for the z0 band using files from the NERSC_TED directory, specifying fiberflat, arcfile, output, and QA files. Use the `--test` flag for testing. ```bash desi_bootcalib.py \ --fiberflat /Users/xavier/DESI/NERSC_TED/pix-z0-00000000.fits \ --arcfile /Users/xavier/DESI/NERSC_TED/pix-z0-00000001.fits \ --outfile /Users/xavier/DESI/NERSC_TED/boot_psf-z0.fits \ --qafile /Users/xavier/DESI/NERSC_TED/qa_boot-z0.pdf \ --test ``` -------------------------------- ### Run desi_bootcalib for b0 Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_fulltest.ipynb Execute the `desi_bootcalib.py` script for the b0 band, specifying fiberflat, arcfile, output, and QA files. Use the `--test` flag for testing. ```bash desi_bootcalib.py \ --fiberflat /Users/xavier/DESI/Wavelengths/pix-b0-00000001.fits \ --arcfile /Users/xavier/DESI/Wavelengths/pix-b0-00000000.fits \ --outfile /Users/xavier/DESI/Wavelengths/boot_psf-b0.fits \ --qafile /Users/xavier/DESI/Wavelengths/qa_boot-b0.pdf \ --test ``` -------------------------------- ### Get Image Shape Source: https://github.com/desihub/desispec/blob/main/doc/nb/Overscan.ipynb Retrieves the shape (dimensions) of the image data processed with the Savitzky-Golay filter. ```python img_savgol.pix.shape ``` -------------------------------- ### Get PSF Dimensions Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_firsttry.ipynb Retrieves the number of pixels along the x and y dimensions of the PSF model. ```python desi_psf.npix_x, desi_psf.npix_y ``` -------------------------------- ### Initialize and Fit Fiber Traces Source: https://github.com/desihub/desispec/blob/main/doc/nb/BootCalib_CRs.ipynb Performs a crude initialization of fiber traces and then fits polynomial traces to the identified fiber positions. This involves two steps: crude initialization and polynomial fitting. ```python xset, xerr = desiboot.trace_crude_init(flat,xpk,ypos) xfit, fdicts = desiboot.fit_traces(xset,xerr) ``` -------------------------------- ### Get Wavelength Array for Fiber Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_firsttry.ipynb Retrieves the wavelength array for a specific fiber across all its pixels. ```python wave0 = desi_psf.wavelength(0,np.arange(desi_psf.npix_y)) ``` -------------------------------- ### Initialize and Load QA Production Data Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_Prod.ipynb Initializes the `QA_Prod` object and loads the production QA data. This is a prerequisite for most subsequent analysis steps. ```python reload(dqap) qa_prod = dqap.QA_Prod() ``` ```python qa_prod.load_data() ``` -------------------------------- ### Get Filtered Target IDs Source: https://github.com/desihub/desispec/blob/main/doc/nb/Lya-tsnr-signal.ipynb Retrieves the TARGETIDs from the fiber map that match the target quasar list. ```python fmap_ids = dat['FIBERMAP'].data['TARGETID'][isin] ``` -------------------------------- ### Get Magnitudes Array Size Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_S2N_Develop.ipynb Calculates the total number of elements in the 'MAGNITUDES' array within 's2n_dict'. ```python np.array(s2n_dict['MAGNITUDES']).size ``` -------------------------------- ### Initialize Matching Variables and Loop Source: https://github.com/desihub/desispec/blob/main/doc/nb/Holy_grail.ipynb Initializes arrays for matching spectral line identifications and ions. It then begins a nested loop structure to calculate median and mean offsets between observed and theoretical line positions, preparing for spectral matching. ```python # Match up (ugly loops) ids = np.zeros(aparm['Nstrong']) ids2 = np.zeros(aparm['Nstrong']) ions = np.array(['12345']*aparm['Nstrong']) for kk in range(aparm['Nstrong']): med_off = np.zeros(nlist) mean_off = np.zeros(nlist) for ss in range(nlist): dpix = dpix_list[ss] min_off = [] for jj in range(aparm['Nstrong']): min_off.append(np.min(np.abs(dpix_obs[kk,jj]-dpix))) min_off.sort() mean_off[ss] = np.mean(min_off[:-1]) med_off[ss] = np.median(min_off) #xdb.set_trace() # if (kk == 1) & ((ss == 13) or (ss==8)): ``` -------------------------------- ### Load a line triplet Source: https://github.com/desihub/desispec/blob/main/doc/nb/Holy_grail.ipynb Loads the first line triplet from the line_triplets list. This is a setup step for further analysis. ```python ltrip = line_triplets[0] ``` -------------------------------- ### Import necessary libraries Source: https://github.com/desihub/desispec/blob/main/doc/nb/Lya-tsnr-signal.ipynb Imports essential libraries for data manipulation, calculations, and plotting. Ensure these are installed in your environment. ```python import pandas import numpy as np import pylab as pl import astropy.io.fits as fits import matplotlib.pyplot as plt from astropy.table import Table, join from astropy.convolution import convolve, Gaussian1DKernel from desiutil.dust import mwdust_transmission ``` -------------------------------- ### Get unique TILEIDs from ztile Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Finds and returns an array of unique TILEIDs present in the filtered ztile table. ```python np.unique(ztile['TILEID']) ``` -------------------------------- ### Get Median SNR Array Size Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_S2N_Develop.ipynb Calculates the total number of elements in the 'MEDIAN_SNR' array within 's2n_dict'. ```python np.array(s2n_dict['MEDIAN_SNR']).size ``` -------------------------------- ### Import Libraries for Bootcalib Source: https://github.com/desihub/desispec/blob/main/doc/nb/Bootcalib_r.ipynb Imports necessary libraries for boot calibration, including desispec, desiutil, astropy, and numpy. It also sets up seaborn for plotting. ```python %matplotlib inline ``` ```python # imports try: import seaborn as sns; sns.set(context="notebook",font_scale=2) except: pass from desispec import bootcalib as desiboot from desiutil import funcfits as dufits from astropy.io import fits from astropy.stats import sigma_clip import numpy as np from astropy.modeling import models, fitting ``` -------------------------------- ### Run desi_bootcalib script Source: https://github.com/desihub/desispec/blob/main/doc/nb/Bootstrap_tests.ipynb Executes the desi_bootcalib script with specified input arc and fiberflat files, output PSF file, and QA file. ```bash desi_bootcalib.py \ --fiberflat /Users/xavier/DESI/Wavelengths/pix-sub_b0-00000001.fits \ --arcfile /Users/xavier/DESI/Wavelengths/pix-sub_b0-00000000.fits \ --outfile /Users/xavier/DESI/Wavelengths/boot_psf-sub_b0.fits \ --qafile /Users/xavier/DESI/Wavelengths/qa_boot-sub_b0.pdf ``` -------------------------------- ### Set Per-Login Paths for DESI Coadd Source: https://github.com/desihub/desispec/blob/main/doc/coadd.md Manually set the PATH and PYTHONPATH environment variables for using the desispec installation. ```bash cd desispec export PATH=$PWD/bin:$PATH export PYTHONPATH=$PWD/py:$PYTHONPATH ``` -------------------------------- ### Compute Fiber Flat using command line Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_Frame.ipynb Command-line execution of desi_compute_fiberflat.py script. Requires input frame, output file, fibermap, and QA file paths. ```bash desi_compute_fiberflat.py \ --infile /Users/xavier/DESI/TST/dogwood/exposures/20150211/00000001/frame-b0-00000001.fits \ --outfile /Users/xavier/DESI/TST/dogwood/calib2d/20150211/jxp-fflat-b0-00000001.fits \ --fibermap /Users/xavier/DESI/TST/data/20150211/fibermap-00000002.fits \ --qafile /Users/xavier/DESI/TST/dogwood/exposures/20150211/00000001/qa-b0-00000001.yaml \ --qafig /Users/xavier/DESI/TST/dogwood/exposures/20150211/00000001/qa-fflat-b0-00000001.pdf ``` -------------------------------- ### Get Unique Programs (Tiles) Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Extracts and returns the unique program names directly from the 'PROGRAM' column of the 'tiles_fits' table. ```python np.unique(tiles_fits['PROGRAM']) ``` -------------------------------- ### Get Unique Goal Types (Exposures) Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Extracts and returns the unique goal types from the 'GOALTYPE' column of the 'exposures_fits' table. ```python np.unique(exposures_fits['GOALTYPE']) ``` -------------------------------- ### Get Unique Programs (Exposures) Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Extracts and returns the unique program names directly from the 'PROGRAM' column of the 'exposures_fits' table. ```python np.unique(exposures_fits['PROGRAM']) ``` -------------------------------- ### Initialize Configuration Source: https://github.com/desihub/desispec/blob/main/doc/nb/tsnr_ensemble.ipynb Initializes the configuration dictionary with tracer type and its corresponding SV1 map value. This sets up essential parameters for data reduction. ```python config = {'tracer': tracer_type, 'sv1_tracer': sv1_map[tracer_type]} ``` -------------------------------- ### Initializing the Fitter Source: https://github.com/desihub/desispec/blob/main/doc/nb/Holy_grail.ipynb Sets up the Levenberg-Marquardt least-squares fitter from astropy.modeling.fitting. This fitter is commonly used for non-linear least squares problems. ```python fitter = fitting.LevMarLSQFitter() ``` -------------------------------- ### Get indices of detected peaks Source: https://github.com/desihub/desispec/blob/main/doc/nb/Holy_grail.ipynb Extracts the pixel indices where spectral lines are detected based on the refined peak detection. ```python xpk = np.where(gdp)[0] ``` -------------------------------- ### Get Unique Target IDs Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Combines TARGETID data from both observed and potential target tables and finds the unique values. ```python all_targetids = np.unique(np.hstack((tractorphot_ids['TARGETID'].data, tractorphot_potential_ids['TARGETID']))) ``` -------------------------------- ### Run desi_bootcalib for trace only Source: https://github.com/desihub/desispec/blob/main/doc/nb/Boot_Calibs_fulltest.ipynb Execute `desi_bootcalib.py` with the `--trace_only` flag to process only the trace, specifying fiberflat, output, and QA files. ```bash desi_bootcalib.py \ --fiberflat /Users/xavier/DESI/Wavelengths/pix-b0-00000001.fits \ --outfile /Users/xavier/DESI/Wavelengths/boot_psf_trace_only-b0.fits \ --qafile /Users/xavier/DESI/Wavelengths/qa_boot-b0.pdf \ --test --trace_only ``` -------------------------------- ### Import desispec.io module Source: https://github.com/desihub/desispec/blob/main/doc/nb/Intro_to_DESI_spectra.ipynb Import the necessary I/O module from the desispec library to read spectral data. ```python from desispec import io ``` -------------------------------- ### Get Bounding Box Source: https://github.com/desihub/desispec/blob/main/doc/nb/Cosmics.ipynb Retrieves the current bounding box coordinates (xmin, xmax, ymin, ymax) of the blob group. ```python def getBoundingBox(self): """Get the bounding rectangle of the group.""" return (self.xmin, self.xmax, self.ymin, self.ymax) ``` -------------------------------- ### Download flat file from URL Source: https://github.com/desihub/desispec/blob/main/doc/nb/Bootstrap_tests.ipynb Downloads the flat FITS file from the specified URL and saves it locally with a .gz extension. This is part of setting up for unit tests. ```python f = urllib2.urlopen(url_flat) tst_fil = 'tmp_flat.fits.gz' with open(tst_fil, "wb") as code: code.write(f.read()) ``` -------------------------------- ### Retrieving Wavelength Values Source: https://github.com/desihub/desispec/blob/main/doc/nb/Holy_grail.ipynb Extracts a list of wavelength values from a specific range. Used to get the initial set of wavelengths for fitting. ```python wvval0 = list(gd_lines[icen-2:icen+3]) wvval0 ``` -------------------------------- ### Get Unique Programs from FAFLAVOR (Tiles) Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Extracts and returns the unique program names associated with FAFLAVOR entries in the 'tiles_fits' table. ```python np.unique(faflavor2program(tiles_fits['FAFLAVOR'])) ``` -------------------------------- ### Compute Sky using command line Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_Frame.ipynb Command-line execution of desi_compute_sky.py script. Requires input frame, fibermap, fiberflat, output file, and QA file paths. Output is redirected to tmp.log. ```bash desi_compute_sky.py \ --infile /Users/xavier/DESI/TST/dogwood/exposures/20150211/00000002/frame-b0-00000002.fits \ --fibermap /Users/xavier/DESI/TST/data/20150211/fibermap-00000002.fits \ --fiberflat /Users/xavier/DESI/TST/dogwood/calib2d/20150211/fiberflat-b0-00000001.fits \ --outfile /Users/xavier/DESI/TST/dogwood/exposures/20150211/00000002/sky2-b0-00000002.fits \ --qafile /Users/xavier/DESI/TST/dogwood/exposures/20150211/00000002/qa-b0-00000002.yaml \ --qafig /Users/xavier/DESI/TST/dogwood/exposures/20150211/00000002/qa-b0-00000002.pdf \ > tmp.log ``` -------------------------------- ### Get Unique Programs from FAFLAVOR (Exposures) Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Extracts and returns the unique program names associated with FAFLAVOR entries in the 'exposures_fits' table. ```python np.unique(faflavor2program(exposures_fits['FAFLAVOR'])) ``` -------------------------------- ### Initialize matching arrays Source: https://github.com/desihub/desispec/blob/main/doc/nb/Holy_grail.ipynb Initializes arrays to store the wavelengths and indices of matched spectral lines. These arrays will be populated during the matching process. ```python # mtch_wv = np.zeros( (tcent.size, gdp.size)) mtch_idx = np.zeros( (tcent.size, gdp.size),dtype=int) ``` -------------------------------- ### Set DESI log level and get logger Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Configures the logging level for DESI operations to DEBUG and retrieves the logger instance. ```python os.environ['DESI_LOGLEVEL'] = 'DEBUG' dsr.log = get_logger() ``` -------------------------------- ### Open FITS File and Access 'z3' Extension Source: https://github.com/desihub/desispec/blob/main/doc/nb/Overscan.ipynb Opens a FITS file and accesses the 'z3' extension. Ensure 'fits' and 'ifile' are defined. This prepares the data for further preprocessing. ```python hdulist = fits.open(ifile) #hdulist.info() z3 = hdulist['z3'] ``` -------------------------------- ### Get Science Object Index Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_S2N_Develop.ipynb Finds the index of the 'SCIENCE' object within the list of object types for which S/N metrics were calculated. ```python sci_idx = qa_data['20200316'][21]['b1']['S2N']['METRICS']['OBJLIST'].index('SCIENCE') sci_idx ``` -------------------------------- ### Initialize QA_Frame with 'science' flavor and sky subtraction Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_Frame.ipynb Initializes a QA_Frame object with the 'science' flavor, initializes sky subtraction, and prints the data. ```python reload(qa_exp) qaframe = qa_exp.QA_Frame(flavor='science') qaframe.init_skysub() print(qaframe.data) ``` -------------------------------- ### Create DESI Environment Shell Function Source: https://github.com/desihub/desispec/blob/main/doc/install.md Define a bash function to load DESI software into your environment. Customize the installation path if not at NERSC. ```bash desidev () { # This is the install location of our desi software. # If you are not at NERSC, then change this to something # without "NERSC_HOST" in the name. desisoft="${HOME}/desi-${NERSC_HOST}" # Set environment variables export CPATH=${desisoft}/include:${CPATH} export LIBRARY_PATH=${desisoft}/lib:${LIBRARY_PATH} export LD_LIBRARY_PATH=${desisoft}/lib:${LD_LIBRARY_PATH} export PYTHONPATH=${desisoft}/lib/python3.5/site-packages:${PYTHONPATH} # Special setup for redmonster red="${HOME}/git-${NERSC_HOST}/redmonster" export PYTHONPATH=${red}/python:${PYTHONPATH} export REDMONSTER_TEMPLATES_DIR=${red}/templates # Choose what data files to use- these locations # are for NERSC. export DESI_ROOT=/project/projectdirs/desi export DESIMODEL=${DESI_ROOT}/software/edison/desimodel/master export DESI_BASIS_TEMPLATES=${DESI_ROOT}/spectro/templates/basis_templates/v2.2 export STD_TEMPLATES=${DESI_ROOT}/spectro/templates/star_templates/v1.1/star_templates_v1.1.fits } ``` -------------------------------- ### Construct FITS File Path for 'z3' Source: https://github.com/desihub/desispec/blob/main/doc/nb/Overscan.ipynb Constructs the full path to a FITS file for the 'z3' extension. Ensure 'exp_path' and 'night' are defined. This is similar to constructing paths for other extensions. ```python iid = 20655 ifile = os.path.join(exp_path, night, '000{}'.format(iid), 'desi-000{}.fits.fz'.format(iid)) ifile ``` -------------------------------- ### Load targetphot files into the database Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Initiates the loading process for targetphot files using the configuration defined in the 'loader' list. ```python dsr.load_file(**(loader[0])) ``` -------------------------------- ### Set Package Version from Git Source: https://github.com/desihub/desispec/blob/main/README.rst If you have tagged a version, use this command to set the package version based on your current git location before installing. ```bash python setup.py version ``` -------------------------------- ### Define Target Photometry File Path Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Constructs the file path for potential target photometry data, incorporating environment variables for project and product information. ```python targetphot_potential_file = os.path.join(os.environ['DESI_ROOT'], 'users', os.environ['USER'], 'lsdr9-photometry', os.environ['SPECPROD'], targetphot_version, 'potential-targets', f"targetphot-potential-{os.environ['SPECPROD']}.fits") targetphot_potential = Table.read(targetphot_potential_file, hdu=1) ``` -------------------------------- ### Stacking Spectra Objects Source: https://context7.com/desihub/desispec/llms.txt Illustrates how to combine multiple Spectra objects into a single one. ```APIDOC ## Stacking Spectra Objects ### Description This section describes how to stack multiple `Spectra` objects together. This operation requires that all spectra being stacked share the same bands and wavelength grid. ### Method `stack(list_of_spectra)` ### Parameters - **list_of_spectra** (list of Spectra) - A list containing `Spectra` objects to be combined. ### Request Example ```python # Assuming 'sp' and 'sp2' are Spectra objects with compatible bands and wave grids combined = stack([sp, sp2]) print(combined.num_spectra()) # 8 ``` ### Response #### Success Response (200) - **combined** (Spectra) - A new Spectra object containing the combined spectra from the input list. #### Response Example ``` 8 ``` ``` -------------------------------- ### Initialize a 3x3 NumPy Array Source: https://github.com/desihub/desispec/blob/main/doc/nb/JoinCosmics.ipynb Initializes a 3x3 NumPy array filled with zeros. This can serve as a starting point for various array manipulations. ```python n1 = np.zeros((3,3)) ``` -------------------------------- ### Get Overscan Column Data Source: https://github.com/desihub/desispec/blob/main/doc/nb/Overscan.ipynb Extracts the overscan column data from the raw image using the provided header information. Requires the get_oscan_col function. ```python oscan_col = get_oscan_col(rawimage, header) ``` -------------------------------- ### Create and Write FITS Header with Comments and History Source: https://github.com/desihub/desispec/blob/main/doc/nb/PreprocConstructionTest.ipynb Generates a FITS PrimaryHDU with random data, adds multiple COMMENT and HISTORY keywords, and writes it to a file. This demonstrates how fitsio handles multiple entries for the same keyword. ```python img = fits.PrimaryHDU(np.random.uniform(size=(10,10))) img.header['COMMENT'] = 'This is the first comment.' img.header['COMMENT'] = 'This is the second comment.' img.header['HISTORY'] = 'This is the first history.' img.header['HISTORY'] = 'This is the second history.' img.header['EXTNAME'] = 'TEST' img.header['PLANCK'] = (6.62607015e-34, "[J s] Planck's Constant") hdulist = fits.HDUList([img]) hdulist.writeto(os.path.join(os.environ['CSCRATCH'], 'fitsio_header_test.fits'), overwrite=True) ``` -------------------------------- ### Create and Save Binary Image Source: https://github.com/desihub/desispec/blob/main/doc/nb/JoinCosmics.ipynb Generates a dummy binary image and saves it as a PDF file. Demonstrates the usage of the `plot_binary` function. ```python dummy = np.zeros((28, 28)) dummy[3:4,4:10] = 1 dummy[3:4,12:15] = 1 dummy[5:6,5:12] = 1 dummy[5:6,15:18] = 1 dummy[11:12,6:20] = 1 dummy[11:12,22:24] = 1 dummy[20:21,5:12] = 1 dummy[20:21,14:24] = 1 fig = plot_binary(dummy) fig.savefig('binary_img.pdf') ``` -------------------------------- ### Get shape of DFLUX_B HDU Source: https://github.com/desihub/desispec/blob/main/doc/nb/Lya-tsnr-signal.ipynb Retrieves and displays the shape of the 'DFLUX_B' ImageHDU from the loaded FITS file. This helps in understanding the dimensions of the data array. ```python ens['DFLUX_B'].shape ``` -------------------------------- ### Import necessary libraries Source: https://github.com/desihub/desispec/blob/main/doc/nb/DatabaseConsistencyChecks.ipynb Imports essential libraries for data manipulation, plotting, and specific DESI functionalities. Includes setup for IERS table freezing. ```python %matplotlib inline import os import glob import json from pytz import utc import numpy as np import matplotlib.pyplot as plt import fitsio from astropy.io import fits from astropy.table import Table, Column, MaskedColumn, join from astropy.time import Time from desiutil.iers import freeze_iers from desiutil.log import get_logger, DEBUG, INFO from desispec.io.meta import faflavor2program from desispec.io.util import checkgzip import desispec.database.redshift as dsr from desispec.database.util import targetphotid, surveyid from desitarget.targets import decode_targetid freeze_iers() ``` -------------------------------- ### Get Survey and Program for a Specific Exposure Source: https://github.com/desihub/desispec/blob/main/doc/nb/FibermapCombinationTest.ipynb Finds the survey and program for a specific exposure ID (77952) on a given night (20210224) and returns it as a formatted string. ```python exposures_row = np.nonzero((exposures['NIGHT'] == 20210224) & (exposures['EXPID'] == 77952))[0][0] survey_program = "{0}-{1}".format(exposures['SURVEY'][exposures_row], program[exposures_row]) survey_program ``` -------------------------------- ### Loading Raw Data and Fiberassign Files Source: https://github.com/desihub/desispec/blob/main/doc/nb/FibermapCombinationTest.ipynb Loads raw spectrographic data and its corresponding fiber assignment file. Includes error handling for different header types and attempts to find the fiber assignment file. ```python log = get_logger() rawfile = findfile('raw', night, int(expid)) try: rawheader = fits.getheader(rawfile, 'SPEC', disable_image_compression=True) except KeyError: rawheader = fits.getheader(rawfile, 'SPS', disable_image_compression=True) rawfafile = fafile = find_fiberassign_file(night, int(expid)) ``` -------------------------------- ### Sort spectral lines and get count Source: https://github.com/desihub/desispec/blob/main/doc/nb/Holy_grail.ipynb Sorts the spectral lines by wavelength and returns the total number of lines. This is a common preprocessing step for spectral analysis. ```python llist.sort('wave') nline = len(llist) nline ``` -------------------------------- ### Load QA Exposure data Source: https://github.com/desihub/desispec/blob/main/doc/nb/QA_Exposure.ipynb Initializes the QA_Exposure object by loading data for a specific exposure, night, and type, using the provided spectro production directory and multi-root name. ```python # Load data reload(dqaexp) qaexp = dqaexp.QA_Exposure(expid, night, 'science', specprod_dir=specprod_dir, multi_root=multi_root) ```