### cryoDRGN Workflow Example Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt A comprehensive cryoDRGN workflow example covering downsampling, parsing, validation, training, analysis, filtering, and trajectory generation. ```bash # Step 1: Downsample images for initial training cryodrgn downsample particles.mrcs -D 128 -o particles.128.mrcs ``` ```bash # Step 2: Parse poses and CTF from RELION refinement cryodrgn parse_pose_star refinement.star -o pose.pkl cryodrgn parse_ctf_star refinement.star -o ctf.pkl -D 300 --Apix 1.03 ``` ```bash # Step 3: Validate preprocessing with backprojection cryodrgn backproject_voxel particles.128.mrcs --poses pose.pkl --ctf ctf.pkl \ -o backproject_test/ --first 10000 ``` ```bash # Step 4: Train initial model on downsampled images cryodrgn train_vae particles.128.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 8 -n 25 -o 00_train128 ``` ```bash # Step 5: Analyze results and filter particles if needed cryodrgn analyze 00_train128 25 --Apix 2.06 cryodrgn filter 00_train128 --epoch 25 ``` ```bash # Step 6: Downsample to higher resolution and retrain cryodrgn downsample particles.mrcs -D 256 -o particles.256.mrcs cryodrgn train_vae particles.256.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 8 -n 25 -o 01_train256 --ind 00_train128/indices.pkl ``` ```bash # Step 7: Final analysis cryodrgn analyze 01_train256 25 --Apix 1.03 --ksample 50 ``` ```bash # Step 8: Generate trajectories for movies cryodrgn graph_traversal 01_train256/z.25.pkl \ --anchors 01_train256/analyze.25/kmeans50/centers_ind.txt \ -o trajectories/z-path.txt ``` ```bash cryodrgn eval_vol 01_train256/weights.25.pkl -c 01_train256/config.yaml \ --zfile trajectories/z-path.txt -o trajectories/volumes/ --Apix 1.03 ``` -------------------------------- ### Install cryoDRGN Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Commands to set up a conda environment and install the cryoDRGN package. ```bash # Create and activate conda environment conda create --name cryodrgn python=3.13 conda activate cryodrgn # Install stable release pip install cryodrgn # Or install beta release for latest features pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ cryodrgn --pre ``` -------------------------------- ### Install cryoDRGN in Developer Mode Source: https://github.com/ml-struct-bio/cryodrgn/wiki/Developer-Setup After cloning the repository and checking out the 'develop' branch, use these commands to create a new feature branch and install cryoDRGN with development dependencies. ```bash cd git checkout -b pip install -e .[dev] ``` -------------------------------- ### K-means clustering setup Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_filtering_template.ipynb Optional re-running of K-means clustering. ```python # Optionally, re-run kmeans with the desired number of classes #K = 20 #kmeans_labels, kmeans_centers = analysis.cluster_kmeans(z, K) ``` -------------------------------- ### Install cryoDRGN development version Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Install a newer, less stable development version of cryoDRGN using the beta release channel on test.pypi.org. This requires specifying the beta index URL. ```bash (cryodrgn) $ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ cryodrgn --pre ``` -------------------------------- ### Analyze CryoDRGN Results Example Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Example usage of the `cryodrgn analyze` command to analyze results from a specified directory after a certain number of training epochs, including setting the pixel size. ```bash $ cryodrgn analyze 01_cryodrgn256 25 --Apix 1.31 # 25 for 1-based indexing of epoch numbers ``` -------------------------------- ### Train cryoDRGN Model (Full Resolution) Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Example command to train a cryoDRGN model for 25 epochs on full-resolution images. Recommended after validating results on downsampled data. ```bash cryodrgn train_vae particles.256.mrcs \ --poses pose.pkl \ --ctf ctf.pkl \ --zdim 8 -n 25 \ -o 01_cryodrgn256 ``` -------------------------------- ### Install Pre-commit Hook Source: https://github.com/ml-struct-bio/cryodrgn/wiki/Developer-Setup Install the pre-commit hook to automatically check for code style and formatting issues before each commit. This helps maintain code quality. ```bash pre-commit install ``` -------------------------------- ### Install cryoDRGN via pip Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Install the stable version of cryoDRGN using pip in a conda environment. Ensure Python 3.10-3.13 is used. ```bash (base) $ conda create --name cryodrgn python=3.13 (cryodrgn) $ conda activate cryodrgn (cryodrgn) $ pip install cryodrgn ``` -------------------------------- ### Train cryoDRGN Model (Downsampled Images) Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Example command to train a cryoDRGN model for 25 epochs on a downsampled image dataset. Use this for initial sanity checks and particle filtering. ```bash cryodrgn train_vae particles.128.mrcs \ --poses pose.pkl \ --ctf ctf.pkl \ --zdim 8 -n 25 \ -o 00_cryodrgn128 ``` -------------------------------- ### Extend cryoDRGN Model Training Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Example command to extend the training of a previously trained cryoDRGN model to a total of 50 epochs. Use the `--load` argument with the path to the weights from the previous training. ```bash cryodrgn train_vae particles.256.mrcs \ --poses pose.pkl \ --ctf ctf.pkl \ --zdim 8 -n 50 \ -o 01_cryodrgn256 \ --load 01_cryodrgn256/weights.25.pkl ``` -------------------------------- ### Setup Grid Plot for Landscape Analysis Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb Initializes a figure and a GridSpec object for creating a grid of plots, likely to visualize pairwise relationships between principal components of the volume data. The number of principal components to display is configurable. ```python # Grid plot landscape -- energy scale # Set up the triangular grid layout n_pcs = 5 # CHANGE ME IF NEEDED fig = plt.figure(figsize=(15, 15)) gs = gridspec.GridSpec(n_pcs-1, n_pcs-1, wspace=0, hspace=0) ``` -------------------------------- ### Import Libraries for CryoDRGN Analysis Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Imports essential Python libraries for data manipulation, visualization, and CryoDRGN-specific functionalities. Ensure these libraries are installed in your environment. ```python import pandas as pd import numpy as np import pickle import subprocess import os, sys from cryodrn import analysis from cryodrgn import utils from cryodrgn import dataset from cryodrgn import ctf from cryodrgn import source import cryodrgn.config from cryodrgn.dataset import TiltSeriesData import matplotlib.pyplot as plt import seaborn as sns import plotly.graph_objs as go import plotly.offline as py from ipywidgets import interact, interactive, HBox, VBox from scipy.spatial.transform import Rotation as RR py.init_notebook_mode() from IPython.display import FileLink, FileLinks ``` -------------------------------- ### Train VAE with cryodrgn Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Trains a variational autoencoder for heterogeneous reconstruction. Supports resuming training from checkpoints and multi-GPU setups. Custom architectures can be specified. ```bash cryodrgn train_vae particles.128.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 8 -n 25 -o 00_cryodrgn128 ``` ```bash cryodrgn train_vae particles.256.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 8 -n 25 -o 01_cryodrgn256 ``` ```bash cryodrgn train_vae particles.256.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 8 -n 50 -o 01_cryodrgn256 --load 01_cryodrgn256/weights.25.pkl ``` ```bash cryodrgn train_vae particles.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 8 -n 50 -o output_dir --load latest ``` ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 cryodrgn train_vae particles.256.mrcs \ --poses pose.pkl --ctf ctf.pkl --zdim 8 -n 25 -o output_dir --multigpu ``` ```bash cryodrgn train_vae particles.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 10 -n 30 -o output_dir \ --enc-layers 4 --enc-dim 1024 --dec-layers 4 --dec-dim 1024 ``` ```bash cryodrgn train_vae particles.mrcs --poses pose.pkl --ctf ctf.pkl \ --zdim 8 -n 25 -o output_dir --do-pose-sgd --domain hartley ``` ```bash cryodrgn train_vae particles_from_M.star --datadir particleseries \ --ctf ctf.pkl --poses pose.pkl --encode-mode tilt \ --dose-per-tilt 2.93 --zdim 12 --num-epochs 50 --beta 0.025 \ -o cryodrgn-et-output ``` -------------------------------- ### Generate a trajectory of volumes Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Create a series of volumes representing a trajectory by specifying `--z-start`, `--z-end`, and `-n` for the number of structures. This example assumes a 1D latent variable model. ```bash $ cryodrgn eval_vol [YOUR_WORKDIR]/weights.pkl --config [YOUR_WORKDIR]/config.yaml -o [WORKDIR]/trajectory \ --z-start -3 --z-end 3 -n 20 ``` -------------------------------- ### Helper Function to Get Output Directory Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb A helper function that creates and returns a unique directory name for saving generated volumes. It checks for existing directories and appends a number to ensure uniqueness. ```python def get_outdir(): '''Helper function to get a clean directory to save volumes''' for i in range(100000): outdir = f'reconstruct_{i:06d}' if os.path.exists(outdir): continue else: break return outdir ``` -------------------------------- ### Import dependencies Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Initializes the notebook environment and imports necessary libraries for data analysis and visualization. ```python import numpy as np import pickle import os from cryodrgn import analysis from cryodrgn import utils from cryodrgn import dataset from cryodrgn import ctf from cryodrgn import config import matplotlib.pyplot as plt import plotly.offline as py from ipywidgets import VBox from scipy.spatial.transform import Rotation as RR py.init_notebook_mode() from IPython.display import FileLinks ``` -------------------------------- ### Get ChimeraX Colors Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Retrieve default color mapping for ChimeraX visualization. ```python # Default chimerax color map colors = analysis._get_chimerax_colors(KMEANS) ``` -------------------------------- ### Get Current Working Directory Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Prints the current working directory. Useful for debugging file paths. ```python pwd ``` -------------------------------- ### Load configuration and model Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Loads the configuration and a trained VAE model from specified files. Requires PyTorch and CryoDRGN libraries. Sets the model to evaluation mode and specifies the device (e.g., CUDA). ```python import numpy as np import torch from cryodrgn import config from cryodrgn.models import HetOnlyVAE from cryodrgn.source import write_mrc # Load configuration and model cfg = config.load("workdir/config.yaml") model, lattice = HetOnlyVAE.load(cfg, "workdir/weights.25.pkl", device=torch.device("cuda")) model.eval() ``` -------------------------------- ### Load Configuration File Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Loads the project configuration from a YAML file. The configuration contains dataset and model parameters necessary for analysis. ```python # Load configuration file config = cryodrgn.config.load(f'{WORKDIR}/config.yaml') print(config) ``` -------------------------------- ### Configure Analysis Paths Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Set the working directory and epoch index for loading data. ```python # Specify the workdir and the epoch number (1-based index) to analyze WORKDIR = '..' EPOCH = None # change me if necessary! ``` -------------------------------- ### CryoDRGN Analyze Command Help Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Displays the help message for the `cryodrgn analyze` command, outlining its usage and available options for visualizing latent space and generating volumes. ```bash usage: cryodrgn analyze [-h] [--device DEVICE] [-o OUTDIR] [--skip-vol] [--skip-umap] [--Apix APIX] [--flip] [--invert] [-d DOWNSAMPLE] [--pc PC] [--ksample KSAMPLE] workdir epoch Visualize latent space and generate volumes positional arguments: workdir Directory with cryoDRGN results epoch Epoch number N to analyze (1-based indexing, corresponding to z.N.pkl, weights.N.pkl) optional arguments: -h, --help show this help message and exit --device DEVICE Optionally specify CUDA device -o OUTDIR, --outdir OUTDIR Output directory for analysis results (default: [workdir]/analyze.[epoch]) --skip-vol Skip generation of volumes --skip-umap Skip running UMAP Extra arguments for volume generation: --Apix APIX Pixel size to add to .mrc header (default: 1 A/pix) --flip Flip handedness of output volumes --invert Invert contrast of output volumes -d DOWNSAMPLE, --downsample DOWNSAMPLE Downsample volumes to this box size (pixels) --pc PC Number of principal component traversals to generate (default: 2) --ksample KSAMPLE Number of kmeans samples to generate (default: 20) ``` -------------------------------- ### Generate Multiple Volumes Along a Trajectory Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Generates multiple volumes by interpolating between start and end points along a trajectory. Requires a trained model and lattice information. ```python z_start = np.array([-2.0] * 8) z_end = np.array([2.0] * 8) for i, alpha in enumerate(np.linspace(0, 1, 10)): z_interp = z_start + alpha * (z_end - z_start) vol = model.decoder.eval_volume( lattice.coords, lattice.D, lattice.extent, norm, z_interp ) write_mrc(f"trajectory_{i:03d}.mrc", vol.cpu().numpy().astype(np.float32), Apix=1.5) ``` -------------------------------- ### Set working directory and epoch Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_filtering_template.ipynb Defines the path to the project directory and the specific epoch to analyze. ```python WORKDIR = '..' EPOCH = None # change me if necessary! ``` -------------------------------- ### CryoDRGN Filter Command Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Demonstrates how to initiate particle filtering using the `cryodrgn filter` command, which can be run interactively from the command line. ```bash cryodrgn filter 01_cryodrgn256 ``` -------------------------------- ### Get Unique Output Directory for Volumes Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Retrieves a unique output directory name using the `get_outdir` helper function and prints its absolute path. This directory will be used to save the generated volumes. ```python # Get a unique output directory, or define your own outdir = get_outdir() print(os.path.abspath(outdir)) ``` -------------------------------- ### Configuration Variables for CryoDRGN Analysis Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb Sets up configuration variables for CryoDRGN analysis, such as epoch, working directory, number of clusters, and linkage method. These variables are essential for directing the analysis and loading the correct data files. ```python EPOCH = None # change me if necessary! WORKDIR = None # Directory with cryoDRGN outputs K = None # Number of sketched volumes M = None # Number of clusters linkage = None # Linkage method used for clustering ``` -------------------------------- ### Initialize Volume Generation Indices Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Initializes an empty list to store indices for volume generation. Users should add specific particle indices to this list. ```python vol_ind = [] # ADD INDICES HERE print(vol_ind) ``` -------------------------------- ### Verify parameters with backprojection Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Perform a voxel-based backprojection to validate that poses and CTF parameters were parsed correctly. ```bash $ cryodrgn backproject_voxel projections.128.mrcs \ --poses pose.pkl \ --ctf ctf.pkl \ -o backproject.128 \ --first 10000 ``` -------------------------------- ### Configure analysis paths Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Sets the working directory and epoch index for loading model results. ```python # Specify the workdir and the epoch number (1-based index) to analyze WORKDIR = '..' EPOCH = None # change me if necessary! print(os.path.abspath(WORKDIR)) ``` -------------------------------- ### Load image dataset Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_filtering_template.ipynb Initializes the image dataset object for particle processing. ```python # load input particles; we can look at the particles = dataset.ImageDataset( config['dataset_args']['particles'], lazy=True, ind=ind_orig, datadir=config['dataset_args']['datadir'] ) N_orig = particles.src.orig_n # particles object can be filtered manually as well # (e.g. to retrieve individual particles) # if ind_orig is not None: # print(f'Filtering particles from {len(particles)} to {len(ind_orig)}') # particles = [particles[int(i)][0, ...] for i in ind_orig] ``` -------------------------------- ### Enable interactive widgets Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_filtering_template.ipynb Configures the Jupyter environment to support interactive widgets. ```bash !jupyter nbextension enable --py widgetsnbextension ``` -------------------------------- ### Accelerated Training with Multiple GPUs Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Demonstrates using the `--multigpu` flag for parallelized training across detected GPUs. The environmental variable `CUDA_VISIBLE_DEVICES` can be used to select specific GPUs. ```bash cryodrgn train_vae ... # Run on GPU 0 ``` ```bash cryodrgn train_vae ... --multigpu # Run on all GPUs on the machine ``` ```bash CUDA_VISIBLE_DEVICES=0,3 cryodrgn train_vae ... --multigpu # Run on GPU 0,3 ``` -------------------------------- ### Enable interactive widgets Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Enables the Jupyter notebook extension required for interactive widgets. ```bash # Enable interactive widgets !jupyter nbextension enable --py widgetsnbextension ``` -------------------------------- ### Import Dependencies Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Initial imports required for cryoDRGN analysis and plotting. ```python from cryodrgn import analysis from cryodrgn import utils import numpy as np import matplotlib.pyplot as plt import seaborn as sns ``` -------------------------------- ### Initialize Interactive Selection for Subset Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Initializes the interactive lasso tool for a subset of particles. This is useful for refining selections or focusing on specific groups. ```python widget, fig, ind_table = analysis.ipy_plot_interactive(df_sub) VBox((widget,fig,ind_table)) ``` -------------------------------- ### Import dependencies Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_filtering_template.ipynb Imports necessary libraries for data manipulation, visualization, and CryoDRGN analysis. ```python import pandas as pd import numpy as np import pickle import subprocess import os, sys from cryodrgn import analysis from cryodrgn import utils from cryodrgn import dataset from cryodrgn import ctf import cryodrgn.config import matplotlib.pyplot as plt import seaborn as sns import plotly.graph_objs as go import plotly.offline as py from ipywidgets import interact, interactive, HBox, VBox from scipy.spatial.transform import Rotation as RR py.init_notebook_mode() from IPython.display import FileLink, FileLinks ``` -------------------------------- ### Plot Learning Curve Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Visualize the training loss over epochs. ```python loss = analysis.parse_loss(f'{WORKDIR}/run.log') plt.figure(figsize=(4, 4)) plt.plot(loss) plt.xlabel("Epoch") plt.ylabel("Loss") plt.axvline(x=EPOCH, linestyle="--", color="black", label=f"Epoch {EPOCH}") plt.legend() plt.tight_layout() #plt.savefig(f"{WORKDIR}/analyze.{EPOCH}/learning_curve_epoch{EPOCH}.png") ``` -------------------------------- ### Parse Poses from Metadata Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Extract rotation and translation parameters from RELION or cryoSPARC files. ```bash # Basic pose extraction from RELION star file cryodrgn parse_pose_star particles.star -o pose.pkl # Override image parameters if not in star file cryodrgn parse_pose_star particles.star -o pose.pkl -D 300 --Apix 1.03 # Parse poses from cryoSPARC .cs file (use parse_pose_csparc) cryodrgn parse_pose_csparc cryosparc_P27_J3_005_particles.cs -o pose.pkl -D 300 ``` -------------------------------- ### Load and Filter CTF Parameters Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Loads Contrast Transfer Function (CTF) parameters from a pickle file and filters them to correspond to the selected representative tilt image for each particle. ```python # Load CTF ctf_params = utils.load_pkl(config['dataset_args']['ctf']) # Filter CTF to one representative tilt for each particle ctf_params = ctf_params[first_tilt_ind] ``` -------------------------------- ### Initialize Interactive Lasso Selection Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Initializes the interactive lasso tool for particle selection. The table widget updates dynamically with the selected indices. Double-click to clear the selection. ```python widget, fig, ind_table = analysis.ipy_plot_interactive(df) VBox((widget,fig,ind_table)) ``` -------------------------------- ### Generate Volumes with Custom Parameters Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Generates 3D volumes using the `generate_volumes` helper function. Allows customization of parameters such as pixel size (Apix), hand flipping, contrast inversion, downsampling, and CUDA device specification. ```python # Modify any defaults for volume generation -- see `cryodrgn eval_vol -h` for details Apix = 1 # Set to volume pixel size flip = False # Hand flip? invert = False # Invert contrast? downsample = None # Set to smaller box size if desired cuda = None # specify cuda device generate_volumes(z[vol_ind], outdir, Apix=Apix, flip=flip, downsample=downsample, invert=invert) ``` -------------------------------- ### Run PCA Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Perform Principal Component Analysis on the latent space. ```python pc, pca = analysis.run_pca(z) ``` -------------------------------- ### Define Landscape and Clustering Directories Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb Constructs paths to the landscape and clustering analysis subdirectories based on configuration variables. These paths are used to load intermediate analysis results. ```python # landscape analysis directory landscape_dir = f'{WORKDIR}/landscape.{EPOCH}' # subdirectories with clustering analysis and volume mapping clustering_dir = os.path.join(landscape_dir, f"sketch_clustering_{linkage}_{M}") landscape_full_dir = os.path.join(landscape_dir, "landscape_full") ``` -------------------------------- ### CryoDRGN eval_vol command help Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Displays the usage and available arguments for the `cryodrgn eval_vol` command, which is used to generate volumes from a trained CryoDRGN model. ```bash usage: cryodrgn eval_vol [-h] -c PKL -o O [--prefix PREFIX] [-v] [-z [Z [Z ...]]] [--z-start [Z_START [Z_START ...]]] [--z-end [Z_END [Z_END ...]]] [-n N] [--zfile ZFILE] [--Apix APIX] [--flip] [-d DOWNSAMPLE] [--norm NORM NORM] [-D D] [--enc-layers QLAYERS] [--enc-dim QDIM] [--zdim ZDIM] [--encode-mode {conv,resid,mlp,tilt}] [--dec-layers PLAYERS] [--dec-dim PDIM] [--enc-mask ENC_MASK] [--pe-type {geom_ft,geom_full,geom_lowf,geom_nohighf,linear_lowf,none}] [--pe-dim PE_DIM] [--domain {hartley,fourier}] [--l-extent L_EXTENT] [--activation {relu,leaky_relu}] weights Evaluate the decoder at specified values of z positional arguments: weights Model weights optional arguments: -h, --help show this help message and exit -c YAML, --config YAML CryoDRGN config.yaml file -o O Output .mrc or directory --prefix PREFIX Prefix when writing out multiple .mrc files (default: vol_) -v, --verbose Increase verbosity Specify z values: -z [Z [Z ...]] Specify one z-value --z-start [Z_START [Z_START ...]] Specify a starting z-value --z-end [Z_END [Z_END ...]] Specify an ending z-value -n N Number of structures between [z_start, z_end] --zfile ZFILE Text file with z-values to evaluate Volume arguments: --Apix APIX Pixel size to add to .mrc header (default: 1 A/pix) --flip Flip handedness of output volume -d DOWNSAMPLE, --downsample DOWNSAMPLE Downsample volumes to this box size (pixels) Overwrite architecture hyperparameters in config.yaml: --norm NORM NORM -D D Box size --enc-layers QLAYERS Number of hidden layers --enc-dim QDIM Number of nodes in hidden layers --zdim ZDIM Dimension of latent variable --encode-mode {conv,resid,mlp,tilt} Type of encoder network --dec-layers PLAYERS Number of hidden layers --dec-dim PDIM Number of nodes in hidden layers --enc-mask ENC_MASK Circular mask radius for image encoder --pe-type {geom_ft,geom_full,geom_lowf,geom_nohighf,linear_lowf,none} Type of positional encoding --pe-dim PE_DIM Num sinusoid features in positional encoding (default: D/2) --domain {hartley,fourier} --l-extent L_EXTENT Coordinate lattice size --activation {relu,leaky_relu} Activation (default: relu) ``` -------------------------------- ### Perform Ab Initio Reconstruction with cryodrgn Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Performs ab initio reconstruction without initial pose estimates, useful when consensus poses are unreliable. Supports multi-GPU training, resuming from checkpoints, and homogeneous reconstruction. ```bash cryodrgn abinit particles.mrcs -o cryodrgn-outs/001_abinit --zdim 8 --ctf ctf.pkl -n 50 ``` ```bash cryodrgn abinit particles.mrcs -o output_abinit --zdim 8 --ctf ctf.pkl \ --num-epochs 50 --epochs-pose-search 5 --l-start 12 --l-end 32 ``` ```bash cryodrgn abinit particles.mrcs -o output_abinit --zdim 8 --ctf ctf.pkl \ -n 50 --multigpu --batch-size-hps 32 --batch-size-sgd 256 ``` ```bash cryodrgn abinit particles.mrcs -o output_abinit --zdim 8 --ctf ctf.pkl \ -n 100 --load output_abinit/weights.50.pkl ``` ```bash cryodrgn abinit particles.mrcs -o output_homo --zdim 0 --ctf ctf.pkl -n 30 ``` ```bash cryodrgn abinit particles.mrcs -o output_abinit --zdim 8 --ctf ctf.pkl \ -n 50 --lazy --datadir /path/to/mrcs/ ``` -------------------------------- ### Load Particle Images Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Loads the particle images using `source.ImageSource.from_file`. It is configured to load lazily and uses the `first_tilt_ind` to select only one representative tilt image per particle. ```python # Load input particles -- only load one tilt image per particle ("ind0") particles = source.ImageSource.from_file(config['dataset_args']['particles'], lazy=True, datadir=config['dataset_args']['datadir'], indices=first_tilt_ind) ``` -------------------------------- ### Load CTF parameters Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Loads and filters CTF parameters if available in the configuration. ```python # Load CTF if "ctf" in config['dataset_args'] and config['dataset_args']['ctf'] is not None: ctf_params = utils.load_pkl(config['dataset_args']['ctf']) if ind_orig is not None: print(f'Filtering ctf parameters from {len(ctf_params)} to {len(ind_orig)}') ctf_params = ctf_params[ind_orig] ctf.print_ctf_params(ctf_params[0]) else: ctf_params = None ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/ml-struct-bio/cryodrgn/wiki/Developer-Setup Use this command to create a new conda environment named 'cryodrgn' with Python 3.9 and activate it. This is a prerequisite for development. ```bash conda create --name cryodrgn python=3.9 && conda activate cryodrgn ``` -------------------------------- ### Create Directory for Volume PCs (Shell Command) Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb A shell command to create a directory named 'volume_pcs'. This is a preparatory step for saving processed volume PCA data. ```shell ''' mkdir volume_pcs ''' ``` -------------------------------- ### Create Subdirectories for Volume PCs (Shell Command) Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb Shell commands to create numbered subdirectories (pc1 to pc5) within the 'volume_pcs' directory. This structure is used to organize the principal components of the volumes. ```shell ''' !for i in {1..5}; do mkdir volume_pcs/pc$i; done ''' ``` -------------------------------- ### Generate Triangular Grid of PCA Clusters Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb Sets up a triangular grid layout to visualize PCA clusters with background scatter points. ```python # Set up the triangular grid layout n_pcs = 5 # CHANGE ME if needed fig = plt.figure(figsize=(15, 15)) gs = gridspec.GridSpec(n_pcs-1, n_pcs-1, wspace=0, hspace=0) # Define the color map for cluster labels cmap = 'tab20' # Loop over each subplot location in the triangular grid for i in range(1, n_pcs): for j in range(i): ax = fig.add_subplot(gs[i-1, j]) # Plot background scatter with light gray points ax.scatter(vol_pc_all[:, j], vol_pc_all[:, i], color='lightgrey', s=1, alpha=0.1, rasterized=True) # Overlay labeled scatter points with color coding sc = ax.scatter(vol_pc[:, j], vol_pc[:, i], c=state_labels, cmap=cmap, s=25, edgecolor='white', linewidths=0.25) # Only set labels for leftmost and bottom plots if j == 0: ax.set_ylabel(f'Volume PC{i+1} (EV: {vol_pca.explained_variance_ratio_[i]:.0%})', fontsize=14, fontweight='bold') if i == n_pcs-1: ax.set_xlabel(f'Volume PC{j+1} (EV: {vol_pca.explained_variance_ratio_[j]:.0%})', fontsize=14, fontweight='bold') # Remove ticks for cleaner look ax.xaxis.set_major_locator(ticker.NullLocator()) ax.yaxis.set_major_locator(ticker.NullLocator()) ``` -------------------------------- ### Save Volume PCs to MRC Files (Commented Out) Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb This commented-out Python code demonstrates how to save the first five principal components of volumes into separate MRC files. It involves reconstructing volumes based on PCA components and saving them into the organized directory structure. ```python # Save first 5 volume PCs ''' for i in range(5): min_, max_ = pc[:,i].min(), pc[:,i].max() print(min_, max_) for j, a in enumerate(np.linspace(min_,max_,10,endpoint=True)): v = volm.copy() v[mask] += pca.components_[i]*a write_mrc(f'volume_pcs/pc{i+1}/{j}.mrc', v) ''' ``` -------------------------------- ### Load model configuration Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Loads the YAML configuration file associated with the trained model. ```python # Load configuration file config = config.load(f'{WORKDIR}/config.yaml') print(config) ``` -------------------------------- ### Import Libraries for CryoDRGN Analysis Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb Imports necessary Python libraries for CryoDRGN landscape analysis, including numerical computation, file handling, plotting, and machine learning. ```python import numpy as np import os from cryodrgn.mrcfile import parse_mrc from cryodrgn import utils import matplotlib.pyplot as plt import seaborn as sns import plotly.offline as py py.init_notebook_mode() import pandas as pd from sklearn.decomposition import PCA from collections import Counter import matplotlib.ticker as ticker import matplotlib.patches as mpatches import matplotlib.gridspec as gridspec ``` -------------------------------- ### Specify Working Directory and Epoch Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Sets the main working directory and the specific epoch number for analysis. The epoch number should be changed if you are analyzing a different training run. ```python # Specify the workdir and the epoch number (1-based index) to analyze WORKDIR = '..' EPOCH = None # change me if necessary! ``` -------------------------------- ### Print CTF Parameters Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Prints the Contrast Transfer Function (CTF) parameters for a given dataset. Ensure `ctf_params` is loaded. ```python ctf.print_ctf_params(ctf_params[0]) ``` -------------------------------- ### Select Subset of Particles for Viewing Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Selects up to 9 particles to view. If more than 9 are available, a random subset is chosen. Otherwise, all available particles are used. ```python # choose 9 particles to view at random if len(particle_ind) > 9: ind_subset9 = np.random.choice(particle_ind, 9, replace=False) else: ind_subset9 = particle_ind print(ind_subset9) ``` -------------------------------- ### Parse CTF Parameters Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Extract contrast transfer function parameters from metadata files. ```bash # Basic CTF extraction with image parameters cryodrgn parse_ctf_star particles.star -o ctf.pkl -D 300 --Apix 1.03 # Override specific CTF parameters cryodrgn parse_ctf_star particles.star -o ctf.pkl -D 300 --Apix 1.03 --kv 300 --cs 2.7 # Generate CTF visualization cryodrgn parse_ctf_star particles.star -o ctf.pkl -D 300 --Apix 1.03 --png ctf_plot.png # Parse CTF from cryoSPARC .cs file (use parse_ctf_csparc) cryodrgn parse_ctf_csparc cryosparc_particles.cs -o ctf.pkl ``` -------------------------------- ### Downsample Particle Images Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Commands to reduce particle image box sizes for faster training. ```bash # Basic downsampling to 128x128 pixels cryodrgn downsample particles.mrcs -D 128 -o particles.128.mrcs # Downsample with particle filtering using index file cryodrgn downsample particles.mrcs -D 164 -o particles.164.mrcs --ind chosen_particles.pkl # Downsample from a .star file with external data directory cryodrgn downsample particles.star -D 128 -o particles.128.mrcs --datadir folder_with_subtilts/ # Downsample preserving .star file structure cryodrgn downsample particles.star -D 128 -o particles.128.star --datadir folder_with_subtilts/ # Split output into chunks for large datasets (avoids memory issues) cryodrgn downsample particles.mrcs -D 256 -o particles.256.mrcs --chunk 10000 # Adjust batch size for memory management cryodrgn downsample particles.txt -D 256 -o particles.256.mrcs -b 2000 ``` -------------------------------- ### Generate volumes from z-values in a file Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Generate multiple volumes using z-values provided in a text file specified by `--zfile`. The file should contain an array of z-values compatible with `np.loadtxt`. ```bash $ cryodrgn eval_vol [WORKDIR]/weights.pkl --config [WORKDIR]/config.yaml --zfile zvalues.txt -o [WORKDIR]/trajectory ``` -------------------------------- ### Load PC2 Z-values Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Loads the Z-values for PC2 from a text file. ```python z_pc2 = np.loadtxt('pc2/z_values.txt') ``` -------------------------------- ### Run Unit Tests Source: https://github.com/ml-struct-bio/cryodrgn/wiki/Developer-Setup Execute the unit tests for cryoDRGN using pytest. This is a crucial step to ensure the project is working correctly after making changes or before submitting a pull request. ```bash pytest tests ``` -------------------------------- ### Annotate Particles for Volume Generation Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Displays an interactive widget to annotate particles, likely for selecting which particles will be used for volume generation. The `vol_ind` list should be populated before this step. ```python widget, fig = analysis.ipy_plot_interactive_annotate(df, vol_ind, opacity=.1) VBox((widget,fig)) ``` -------------------------------- ### Perform Voxel-based Backprojection Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Reconstruct volumes for validation of pose and CTF parameters. ```bash # Basic backprojection with CTF and poses cryodrgn backproject_voxel particles.128.mrcs --ctf ctf.pkl --poses pose.pkl -o backproject/ # Use lazy loading for large datasets to avoid out-of-memory errors cryodrgn backproject_voxel particles.256.mrcs --ctf ctf.pkl --poses pose.pkl \ --ind good_particles.pkl -o backproject-256/ --lazy # Quick test with only first 10000 images cryodrgn backproject_voxel particles.196.mrcs --ctf ctf.pkl --poses pose.pkl \ -o backproject-196/ --lazy --first 10000 # Backproject tilt series data (cryo-ET) cryodrgn backproject_voxel particles_from_M.star --datadir subtilts/128/ \ --ctf ctf.pkl --poses pose.pkl -o backproject-tilt/ --lazy \ --tilt --ntilts 5 --dose-per-tilt 2.93 ``` -------------------------------- ### Load particle dataset Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_viz_template.ipynb Initializes the ImageDataset object to access particle information. ```python # load input particles, first time just to get total number of particles particles = dataset.ImageDataset( config['dataset_args']['particles'], lazy=True, ind=ind_orig, datadir=config['dataset_args']['datadir'] ) N_orig = particles.src.orig_n ``` -------------------------------- ### Load and Filter Particle Poses Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Loads particle poses (rotations and translations) either from a pickle file generated during training or from a specified file. It then converts rotation matrices to Euler angles and filters them to include only the first tilt image per particle. ```python # Load poses if config['dataset_args']['do_pose_sgd']: pose_pkl = f'{WORKDIR}/pose.{EPOCH}.pkl' with open(pose_pkl,'rb') as f: rot, trans = pickle.load(f) else: pose_pkl = config['dataset_args']['poses'] rot, trans = utils.load_pkl(pose_pkl) # Convert rotation matrices to euler angles euler = RR.from_matrix(rot).as_euler('zyz', degrees=True) # Filter poses to one representative tilt for each particle rot0 = rot[first_tilt_ind] trans0 = trans[first_tilt_ind] euler0 = euler[first_tilt_ind] ``` -------------------------------- ### Parse Particle and Tilt Image Relationships Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Parses the particle-to-tilt image mapping from the configuration file, assuming the model's encoding mode is 'tilt'. This establishes how particles correspond to specific tilt images. ```python # Tilt series data -- Get a representative tilt image for each particle assert config['model_args']['encode_mode'] == 'tilt' p_to_t, t_to_p = TiltSeriesData.parse_particle_tilt(config['dataset_args']['particles']) print(f"Parsing {config['dataset_args']['particles']}") print(f"Detected {len(t_to_p)} tilt images for {len(p_to_t)} particles") # Selection for the first tilt of each particle first_tilt_ind = np.array([pp[0] for pp in p_to_t]) ``` -------------------------------- ### Print Working Directory Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Prints the absolute path of the current working directory. This is useful for verifying the WORKDIR setting. ```python print(os.path.abspath(WORKDIR)) ``` -------------------------------- ### Load PC1 Z-values Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Loads the Z-values for PC1 from a text file. ```python z_pc1 = np.loadtxt('pc1/z_values.txt') ``` -------------------------------- ### Load K-means Clustering Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_figures_template.ipynb Load K-means cluster centers index. ```python # Load points KMEANS = None kmeans_ind = np.loadtxt( f'{WORKDIR}/analyze.{EPOCH}/kmeans{KMEANS}/centers_ind.txt', dtype=int ) ``` -------------------------------- ### Analyze cryodrgn Results Source: https://context7.com/ml-struct-bio/cryodrgn/llms.txt Visualizes the learned latent space and generates representative 3D volumes. Supports options to control k-means sampling, low-pass filtering, cropping, and skipping computations like UMAP or volume generation. ```bash cryodrgn analyze 01_cryodrgn256 25 --Apix 1.31 ``` ```bash cryodrgn analyze my_workdir 50 --ksample 50 ``` ```bash cryodrgn analyze my_workdir 50 --low-pass 4 --crop 96 ``` ```bash cryodrgn analyze my_workdir 25 --skip-vol ``` ```bash cryodrgn analyze my_workdir 25 --skip-umap ``` ```bash cryodrgn analyze my_workdir 25 -o custom_analysis_dir ``` ```bash cryodrgn analyze my_workdir 25 --Apix 1.5 --flip ``` ```bash cryodrgn analyze my_workdir 25 --Apix 1.5 -d 128 ``` -------------------------------- ### Plot PCA Components (Hexbin) Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_ET_viz_template.ipynb Generates a hexbin plot of the first two principal components (PC1 vs PC2). Includes error handling for small datasets. Assumes `pc` is available. ```python try: g = sns.jointplot(x=pc[:,0], y=pc[:,1], kind='hex') except ZeroDivisionError: print("Data too small to produce hexbins!") g.set_axis_labels('PC1', 'PC2') ``` -------------------------------- ### Parse image poses Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Extract pose information from consensus reconstruction files into a binary pickle format. ```bash $ cryodrgn parse_pose_star particles.star -o pose.pkl ``` ```bash $ cryodrgn parse_pose_csparc cryosparc_P27_J3_005_particles.cs -o pose.pkl -D 300 ``` -------------------------------- ### CLI: train_vae Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md Command to train a VAE for heterogeneous reconstruction with known poses. ```APIDOC ## CLI train_vae ### Description Train a VAE for heterogeneous reconstruction with known pose. ### Parameters #### Positional Arguments - **particles** (string) - Required - Input particles (.mrcs, .star, .cs, or .txt) #### Optional Arguments - **-o, --outdir** (string) - Required - Output directory to save model - **--zdim** (int) - Required - Dimension of latent variable - **--poses** (string) - Required - Image poses (.pkl) - **--ctf** (string) - Optional - CTF parameters (.pkl) - **--load** (string) - Optional - Initialize training from a checkpoint - **--num-epochs** (int) - Optional - Number of training epochs (default: 20) - **--batch-size** (int) - Optional - Minibatch size (default: 8) - **--lr** (float) - Optional - Learning rate in Adam optimizer (default: 0.0001) - **--multigpu** (flag) - Optional - Parallelize training across all detected GPUs - **--do-pose-sgd** (flag) - Optional - Refine poses with gradient descent ``` -------------------------------- ### Rerun Clustering (Commented Out) Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/cryodrgn/templates/cryoDRGN_analyze_landscape_template.ipynb A commented-out code snippet showing how to rerun clustering using AgglomerativeClustering from scikit-learn. This allows for experimenting with different clustering parameters, such as the number of clusters and linkage method. ```python # Rerun clustering ''' cluster = AgglomerativeClustering(n_clusters=10, affinity='euclidean', linkage='average') labels = cluster.fit_predict(vols) ''' ``` -------------------------------- ### CryoDRGN traversal commands Source: https://github.com/ml-struct-bio/cryodrgn/blob/main/README.md These commands generate text files of z-values that can be used with `cryodrgn eval_vol` to create structural trajectories for visualization. ```bash $ cryodrgn pc_traversal -h ``` ```bash $ cryodrgn graph_traversal -h ``` ```bash $ cryodrgn direct_traversal -h ```