### Install emcee from source Source: https://github.com/dfm/emcee/blob/main/docs/user/install.md Install the latest development version of emcee by cloning the GitHub repository and installing it in editable mode. Ensure pip, setuptools, and related build tools are up-to-date first. ```bash python -m pip install -U pip python -m pip install -U setuptools setuptools_scm pep517 git clone https://github.com/dfm/emcee.git cd emcee python -m pip install -e . ``` -------------------------------- ### Install emcee using pip Source: https://github.com/dfm/emcee/blob/main/docs/user/install.md Install the latest stable version of emcee using pip. Ensure pip, setuptools, and related build tools are up-to-date first. ```bash python -m pip install -U pip pip install -U setuptools setuptools_scm pep517 pip install -U emcee ``` -------------------------------- ### Test emcee installation Source: https://github.com/dfm/emcee/blob/main/docs/user/install.md Run unit and integration tests for emcee after installing from source. This requires pytest and h5py to be installed. ```bash python -m pip install -U pytest h5py python -m pytest -v src/emcee/tests ``` -------------------------------- ### HDFBackend Usage Example Source: https://github.com/dfm/emcee/blob/main/docs/user/backends.md Demonstrates how to use the HDFBackend to read existing chain data from a file. ```APIDOC ## HDFBackend Usage Example ### Description This example shows how to instantiate an `HDFBackend` in read-only mode and retrieve the flattened chain. ### Method ```python reader = emcee.backends.HDFBackend("chain_filename.h5", read_only=True) flatchain = reader.get_chain(flat=True) ``` ### Parameters - `filename` (str): The name of the HDF5 file. - `read_only` (bool, optional): If True, prevents accidental overwriting. Defaults to False. ``` -------------------------------- ### Install emcee using conda Source: https://github.com/dfm/emcee/blob/main/docs/user/install.md Install the latest stable version of emcee using conda. It's recommended to update conda first. ```bash conda update conda conda install -c conda-forge emcee ``` -------------------------------- ### Initialize Walker Starting Positions Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/quickstart.ipynb Define the number of walkers and generate random initial positions for each walker in the parameter space. ```python nwalkers = 32 p0 = np.random.rand(nwalkers, ndim) ``` -------------------------------- ### HDF5 Backend for Persistent Storage Source: https://context7.com/dfm/emcee/llms.txt Shows how to use emcee's HDFBackend for persistent storage of MCMC chains in an HDF5 file. This backend supports resumable runs and requires 'h5py' to be installed. Requires emcee, numpy, and HDFBackend. ```python import numpy as np import emcee from emcee.backends import HDFBackend def log_prob(x): return -0.5 * np.sum(x ** 2) nwalkers, ndim = 20, 4 filename = "chain.h5" ``` -------------------------------- ### Configure Matplotlib and Multiprocessing Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/parallel.ipynb Sets up Matplotlib for high-resolution figures and configures the multiprocessing start method. Ensure this is done before any parallel processing begins. ```python %config InlineBackend.figure_format = "retina" from matplotlib import rcParams rcParams["savefig.dpi"] = 100 rcParams["figure.dpi"] = 100 rcParams["font.size"] = 20 import multiprocessing multiprocessing.set_start_method("fork") ``` -------------------------------- ### run_mcmc Source: https://github.com/dfm/emcee/blob/main/docs/user/sampler.md Runs the MCMC sampler for a specified number of steps, iterating the `sample()` method. It can resume from the last state or start from a new initial state. ```APIDOC ## run_mcmc(initial_state, nsteps, **kwargs) ### Description Iterate [`sample()`](#emcee.EnsembleSampler.sample) for `nsteps` iterations and return the result. This method returns the most recent result from [`sample()`](#emcee.EnsembleSampler.sample). ### Parameters - **initial_state**: The initial state or position vector. Can also be `None` to resume from where :func:`run_mcmc` left off the last time it executed. - **nsteps**: The number of steps to run. Other parameters are directly passed to [`sample()`](#emcee.EnsembleSampler.sample). ``` -------------------------------- ### Run Production MCMC Steps Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/quickstart.ipynb Perform the main production run of the MCMC chain for a specified number of steps, starting from the state saved after burn-in. ```python sampler.run_mcmc(state, 10000); ``` -------------------------------- ### Vectorized log-probability evaluation in emcee Source: https://context7.com/dfm/emcee/llms.txt This example shows how to use the `vectorize=True` option in emcee's EnsembleSampler. This is efficient when your log-probability function can process batches of walker positions simultaneously using libraries like NumPy or JAX. The log-probability function must accept an array of shape `(batch_size, ndim)` and return an array of shape `(batch_size,)`. ```python import numpy as np import emcee # Vectorized: receives (N, ndim) array, must return (N,) array def log_prob_vectorized(X): # X shape: (batch_size, ndim) return -0.5 * np.sum(X ** 2, axis=1) nwalkers, ndim = 40, 6 p0 = np.random.randn(nwalkers, ndim) sampler = emcee.EnsembleSampler( nwalkers, ndim, log_prob_vectorized, vectorize=True ) sampler.run_mcmc(p0, 1000, progress=False) flat = sampler.get_chain(discard=100, flat=True) print("Flat chain shape:", flat.shape) # (36000, 6) print("Acceptance:", np.mean(sampler.acceptance_fraction).round(3)) ``` -------------------------------- ### Initialize, Run, and Resume emcee Sampler with HDFBackend Source: https://context7.com/dfm/emcee/llms.txt Demonstrates initializing an emcee sampler with an HDFBackend, running it for a specified number of steps, and then resuming the run to append more steps. Also shows how to access the chain data in read-only mode. ```python import numpy as np import emcee from emcee.backends import HDFBackend # Assume nwalkers, ndim, log_prob are defined filename = "my_chain.h5" backend = HDFBackend(filename, name="run1") backend.reset(nwalkers, ndim) p0 = np.random.randn(nwalkers, ndim) sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, backend=backend) sampler.run_mcmc(p0, 1000, progress=False) print("Saved iteration:", backend.iteration) # 1000 sampler2 = emcee.EnsembleSampler(nwalkers, ndim, log_prob, backend=backend) sampler2.run_mcmc(None, 500, progress=False) # initial_state=None resumes print("Total iteration:", backend.iteration) # 1500 reader = HDFBackend(filename, name="run1", read_only=True) flat = reader.get_chain(discard=200, thin=5, flat=True) print("Flat chain shape:", flat.shape) # (~5200, 4) tau = reader.get_autocorr_time(quiet=True) print("Autocorr times:", tau.round(2)) ``` -------------------------------- ### Initialize and Run Emcee Sampler Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/line.ipynb Initialize the emcee walkers in a small Gaussian ball around the maximum likelihood solution. Run the MCMC sampler for a specified number of steps, providing the log-probability function and initial positions. ```python import emcee pos = soln.x + 1e-4 * np.random.randn(32, 3) nwalkers, ndim = pos.shape sampler = emcee.EnsembleSampler( nwalkers, ndim, log_probability, args=(x, y, yerr) ) sampler.run_mcmc(pos, 5000, progress=True); ``` -------------------------------- ### EnsembleSampler Initialization and run_mcmc Source: https://context7.com/dfm/emcee/llms.txt Demonstrates how to initialize the EnsembleSampler with a log-probability function and run the MCMC simulation for a specified number of steps. It also shows how to retrieve flattened samples and check the acceptance fraction. ```APIDOC ## EnsembleSampler ### Description The central class that manages an ensemble of walkers sampling a log-probability function. It accepts the number of walkers (`nwalkers`), the dimensionality of the parameter space (`ndim`), and the log-probability function (`log_prob_fn`). Optional arguments control the proposal strategy (`moves`), parallelization (`pool`), storage (`backend`), vectorized evaluation (`vectorize`), and named parameters (`parameter_names`). ## EnsembleSampler.run_mcmc ### Description Runs the sampler for `nsteps` steps starting from `initial_state` and returns the final `State`. Passing `initial_state=None` resumes from where the last call ended. All keyword arguments are forwarded to `sample()`. ### Method `EnsembleSampler.run_mcmc(initial_state, nsteps, **kwargs)` ### Parameters - `initial_state` (State or None): The starting state of the sampler. If None, resumes from the last state. - `nsteps` (int): The number of steps to run the sampler. - `**kwargs`: Additional keyword arguments forwarded to `sample()`. ### Request Example ```python import numpy as np import emcee # Define a log-probability function (5-D correlated Gaussian) def log_prob(x, icov): diff = x - np.array([1.0, 2.0, 3.0, 4.0, 5.0]) return -0.5 * np.dot(diff, np.dot(icov, diff)) ndim, nwalkers = 5, 32 cov = np.diag(np.array([0.1, 0.2, 0.3, 0.4, 0.5]) ** 2) icov = np.linalg.inv(cov) # Initialize walkers in a small ball around the truth rng = np.random.default_rng(42) p0 = rng.standard_normal((nwalkers, ndim)) * 0.01 sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, args=[icov]) sampler.run_mcmc(p0, nsteps=2000, progress=True) # Discard burn-in and thin flat_samples = sampler.get_chain(discard=200, thin=5, flat=True) print("Flat chain shape:", flat_samples.shape) # (5888, 5) print("Mean acceptance fraction:", np.mean(sampler.acceptance_fraction)) ``` ### Response #### Success Response Returns the final `State` object after running the MCMC. #### Response Example ``` Flat chain shape: (5888, 5) Mean acceptance fraction: 0.5 ``` ``` -------------------------------- ### Set up MCMC Sampler with HDF5 Backend Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/monitor.ipynb Defines the log probability function, initializes walkers, sets up an HDF5 backend for saving results, and initializes the EnsembleSampler. ```python import emcee import numpy as np np.random.seed(42) # The definition of the log probability function # We'll also use the "blobs" feature to track the "log prior" for each step def log_prob(theta): log_prior = -0.5 * np.sum((theta - 1.0) ** 2 / 100.0) log_prob = -0.5 * np.sum(theta**2) + log_prior return log_prob, log_prior # Initialize the walkers coords = np.random.randn(32, 5) nwalkers, ndim = coords.shape # Set up the backend # Don't forget to clear it in case the file already exists filename = "tutorial.h5" backend = emcee.backends.HDFBackend(filename) backend.reset(nwalkers, ndim) # Initialize the sampler sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, backend=backend) ``` -------------------------------- ### Set Up Gaussian Hyperparameters Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/quickstart.ipynb Set up the dimensionality, mean vector, and covariance matrix for the multi-dimensional Gaussian distribution. ```python ndim = 5 np.random.seed(42) means = np.random.rand(ndim) cov = 0.5 - np.random.rand(ndim**2).reshape((ndim, ndim)) cov = np.triu(cov) cov += cov.T - np.diag(cov.diagonal()) cov = np.dot(cov, cov) ``` -------------------------------- ### Sample with Default StretchMove Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/moves.ipynb Initializes and runs an emcee sampler using the default StretchMove. It then calculates and prints the autocorrelation time of the chain. ```python import emcee np.random.seed(589403) init = np.random.randn(32, 1) nwalkers, ndim = init.shape sampler0 = emcee.EnsembleSampler(nwalkers, ndim, logprob) sampler0.run_mcmc(init, 5000) print( "Autocorrelation time: {0:.2f} steps".format( sampler0.get_autocorr_time()[0] ) ) ``` -------------------------------- ### Print Autocorrelation Time Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/line.ipynb Calculate and print the integrated autocorrelation time for the sampler. This helps determine how many steps are needed for the chain to 'forget' its starting point. ```python tau = sampler.get_autocorr_time() print(tau) ``` -------------------------------- ### Get Number of Available CPUs Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/parallel.ipynb Retrieves the number of CPU cores available on the system using `multiprocessing.cpu_count()`. This information can be useful for understanding expected performance gains from parallelization. ```python from multiprocessing import cpu_count ncpu = cpu_count() print("{0} CPUs".format(ncpu)) ``` -------------------------------- ### Sample with Custom Move Combination Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/moves.ipynb Sets up and runs an emcee sampler using a custom combination of DEMove (80% probability) and DESnookerMove (20% probability). It then calculates and prints the autocorrelation time. ```python np.random.seed(93284) sampler = emcee.EnsembleSampler( nwalkers, ndim, logprob, moves=[ (emcee.moves.DEMove(), 0.8), (emcee.moves.DESnookerMove(), 0.2), ], ) sampler.run_mcmc(init, 5000) print( "Autocorrelation time: {0:.2f} steps".format( sampler.get_autocorr_time()[0] ) ) plt.plot(sampler.get_chain()[:, 0, 0], "k", lw=0.5) plt.xlim(0, 5000) plt.ylim(-5.5, 5.5) plt.title("move: [(DEMove, 0.8), (DESnookerMove, 0.2)]", fontsize=14) plt.xlabel("step number") plt.ylabel("x"); ``` -------------------------------- ### Run MCMC Sampler Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/autocorr.ipynb Initializes and runs an emcee EnsembleSampler for a multimodal density in three dimensions. Requires emcee and numpy. The sampler is configured with 32 walkers, 3 dimensions, and a log probability function. ```python import emcee def log_prob(p): return np.logaddexp(-0.5 * np.sum(p**2), -0.5 * np.sum((p - 4.0) ** 2)) sampler = emcee.EnsembleSampler(32, 3, log_prob) sampler.run_mcmc( np.concatenate( (np.random.randn(16, 3), 4.0 + np.random.randn(16, 3)), axis=0 ), 500000, progress=True, ); ``` -------------------------------- ### emcee.backends.HDFBackend Source: https://context7.com/dfm/emcee/llms.txt A persistent backend that stores the MCMC chain in an HDF5 file on disk using the `h5py` library. It supports resumable runs, chunked/compressed storage, and read-only access. Requires `h5py` to be installed. ```APIDOC ## emcee.backends.HDFBackend — Persistent HDF5 backend Stores the MCMC chain in an HDF5 file on disk using `h5py`. Supports resumable runs, chunked/compressed storage, and read-only access. Requires `pip install h5py`. ```python import numpy as np import emcee from emcee.backends import HDFBackend def log_prob(x): return -0.5 * np.sum(x ** 2) nwalkers, ndim = 20, 4 filename = "chain.h5" ``` -------------------------------- ### Maximum Likelihood Autocorrelation Function Estimation Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/autocorr.ipynb Estimates $\tau$ using a maximum likelihood approach by fitting a celerite GP model to the chain. Requires installation of celerite and autograd. This method is useful for short chains. ```python from scipy.optimize import minimize def autocorr_ml(y, thin=1, c=5.0): # Compute the initial estimate of tau using the standard method init = autocorr_new(y, c=c) z = y[:, ::thin] N = z.shape[1] # Build the GP model tau = max(1.0, init / thin) kernel = terms.RealTerm( np.log(0.9 * np.var(z)), -np.log(tau), bounds=[(-5.0, 5.0), (-np.log(N), 0.0)], ) kernel += terms.RealTerm( np.log(0.1 * np.var(z)), -np.log(0.5 * tau), bounds=[(-5.0, 5.0), (-np.log(N), 0.0)], ) gp = celerite.GP(kernel, mean=np.mean(z)) gp.compute(np.arange(z.shape[1])) # Define the objective def nll(p): # Update the GP model gp.set_parameter_vector(p) # Loop over the chains and compute likelihoods v, g = zip(*(gp.grad_log_likelihood(z0, quiet=True) for z0 in z)) # Combine the datasets return -np.sum(v), -np.sum(g, axis=0) # Optimize the model p0 = gp.get_parameter_vector() bounds = gp.get_parameter_bounds() soln = minimize(nll, p0, jac=True, bounds=bounds) gp.set_parameter_vector(soln.x) # Compute the maximum likelihood tau a, c = kernel.coefficients[:2] tau = thin * 2 * np.sum(a / c) / np.sum(a) return tau # Calculate the estimate for a set of different chain lengths ml = np.empty(len(N)) ml[:] = np.nan for j, n in enumerate(N[1:8]): i = j + 1 thin = max(1, int(0.05 * new[i])) ml[i] = autocorr_ml(chain[:, :n], thin=thin) ``` -------------------------------- ### Basic Usage of EnsembleSampler Source: https://github.com/dfm/emcee/blob/main/docs/index.md Demonstrates how to initialize and run the emcee EnsembleSampler to draw samples from a multi-dimensional Gaussian distribution. Requires numpy and emcee libraries. ```python import numpy as np import emcee def log_prob(x, ivar): return -0.5 * np.sum(ivar * x ** 2) ndim, nwalkers = 5, 100 ivar = 1. / np.random.rand(ndim) p0 = np.random.randn(nwalkers, ndim) sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, args=[ivar]) sampler.run_mcmc(p0, 10000) ``` -------------------------------- ### emcee.backends.HDFBackend Source: https://github.com/dfm/emcee/blob/main/docs/user/backends.md Initializes the HDFBackend for saving and reading chain data to/from an HDF5 file. ```APIDOC ## emcee.backends.HDFBackend ### Description A backend that stores the chain in an HDF5 file using h5py. You must install [h5py](http://www.h5py.org/) to use this backend. ### Parameters - `filename` (str): The name of the HDF5 file where the chain will be saved. - `name` (str, optional): The name of the group where the chain will be saved. Defaults to 'mcmc'. - `read_only` (bool, optional): If `True`, the backend will throw a `RuntimeError` if the file is opened with write access. Defaults to `False`. - `dtype` (optional): The data type for the samples. - `compression` (optional): The compression filter to use for the datasets. - `compression_opts` (optional): The compression options to use. ``` -------------------------------- ### In-Memory Backend for emcee Source: https://context7.com/dfm/emcee/llms.txt Demonstrates the default in-memory Backend for emcee, which stores chain data in RAM. Shows explicit instantiation and usage for storing positions, log-probabilities, and accepted counts. Requires emcee and numpy. ```python import numpy as np import emcee from emcee.backends import Backend def log_prob(x): return -0.5 * np.sum(x ** 2) nwalkers, ndim = 16, 3 p0 = np.random.randn(nwalkers, ndim) # Explicit Backend (identical to default behavior) backend = Backend() sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, backend=backend) sampler.run_mcmc(p0, 500, progress=False) # Reset and re-run from a different starting point sampler.reset() p0_new = np.random.randn(nwalkers, ndim) * 0.1 sampler.run_mcmc(p0_new, 1000, progress=False) print("Iteration:", backend.iteration) # 1000 print("Shape:", backend.shape) # (16, 3) print("Chain shape:", backend.get_chain().shape) # (1000, 16, 3) print("Acceptance:", backend.accepted / backend.iteration) ``` -------------------------------- ### GaussianMove with Isotropic, Diagonal, and Full Covariance Source: https://context7.com/dfm/emcee/llms.txt Illustrates using emcee's GaussianMove with different covariance settings: isotropic (scalar step size), diagonal (per-dimension step sizes), and full covariance matrix. Requires emcee, numpy, and GaussianMove. ```python import numpy as np import emcee from emcee.moves import GaussianMove def log_prob(x): return -0.5 * np.dot(x, x) nwalkers, ndim = 20, 4 p0 = np.random.randn(nwalkers, ndim) * 0.5 # Isotropic proposal with scalar step size move_iso = GaussianMove(cov=0.1) # Diagonal proposal (different step per dimension) move_diag = GaussianMove(cov=np.array([0.05, 0.1, 0.2, 0.4])) # Full covariance proposal cov_matrix = 0.1 * np.eye(ndim) + 0.05 * np.ones((ndim, ndim)) move_full = GaussianMove(cov=cov_matrix) sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, moves=move_diag) sampler.run_mcmc(p0, 2000, progress=False) print("Shape of chain:", sampler.get_chain().shape) # (2000, 20, 4) print("Acceptance fraction:", np.mean(sampler.acceptance_fraction).round(3)) ``` -------------------------------- ### Initialize Ensemble Sampler Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/quickstart.ipynb Initialize the emcee EnsembleSampler with the number of walkers, dimensions, log probability function, and any additional arguments for the log probability function. ```python import emcee sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, args=[means, cov]) ``` -------------------------------- ### Run MCMC with a New Backend Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/monitor.ipynb Initiates an MCMC run using a specified backend. Progress can be displayed. ```python sampler2.run_mcmc(coords, new_backend.iteration, progress=True); ``` -------------------------------- ### Initialize and Run EnsembleSampler for MCMC Source: https://context7.com/dfm/emcee/llms.txt Defines a log-probability function for a 5-D correlated Gaussian, initializes walkers, and runs the MCMC sampler. Demonstrates discarding burn-in and flattening the chain. ```python import numpy as np import emcee # Define a log-probability function (5-D correlated Gaussian) def log_prob(x, icov): diff = x - np.array([1.0, 2.0, 3.0, 4.0, 5.0]) return -0.5 * np.dot(diff, np.dot(icov, diff)) ndim, nwalkers = 5, 32 cov = np.diag(np.array([0.1, 0.2, 0.3, 0.4, 0.5]) ** 2) icov = np.linalg.inv(cov) # Initialize walkers in a small ball around the truth rng = np.random.default_rng(42) p0 = rng.standard_normal((nwalkers, ndim)) * 0.01 sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, args=[icov]) sampler.run_mcmc(p0, nsteps=2000, progress=True) # Discard burn-in and thin flat_samples = sampler.get_chain(discard=200, thin=5, flat=True) print("Flat chain shape:", flat_samples.shape) # (5888, 5) print("Mean acceptance fraction:", np.mean(sampler.acceptance_fraction)) ``` -------------------------------- ### sample Source: https://github.com/dfm/emcee/blob/main/docs/user/sampler.md Advances the MCMC chain as a generator, yielding the state of the ensemble every `thin_by` steps. Supports tuning, storing samples, and progress reporting. ```APIDOC ## sample(initial_state, log_prob0=None, rstate0=None, blobs0=None, iterations=1, tune=False, skip_initial_state_check=False, thin_by=1, thin=None, store=True, progress=False, progress_kwargs=None) ### Description Advance the chain as a generator. Every `thin_by` steps, this generator yields the [`State`](#emcee.State) of the ensemble. Note that several of the `EnsembleSampler` methods return or consume `State` objects: ### Parameters - **initial_state** (*State* or *ndarray* *[**nwalkers* *,* *ndim* *]*) – The initial [`State`](#emcee.State) or positions of the walkers in the parameter space. - **iterations** (*Optional* *[**int* *or* *NoneType* *]*) – The number of steps to generate. `None` generates an infinite stream (requires `store=False`). - **tune** (*Optional* *[**bool* *]*) – If `True`, the parameters of some moves will be automatically tuned. - **thin_by** (*Optional* *[**int* *]*) – If you only want to store and yield every `thin_by` samples in the chain, set `thin_by` to an integer greater than 1. When this is set, `iterations * thin_by` proposals will be made. - **store** (*Optional* *[**bool* *]*) – By default, the sampler stores (in memory) the positions and log-probabilities of the samples in the chain. If you are using another method to store the samples to a file or if you don’t need to analyze the samples after the fact (for burn-in for example) set `store` to `False`. - **progress** (*Optional* *[**bool* *or* *str* *]*) – If `True`, a progress bar will be shown as the sampler progresses. If a string, will select a specific `tqdm` progress bar - most notable is `'notebook'`, which shows a progress bar suitable for Jupyter notebooks. If `False`, no progress bar will be shown. - **progress_kwargs** (*Optional* *[**dict* *]*) – A `dict` of keyword arguments to be passed to the tqdm call. - **skip_initial_state_check** (*Optional* *[**bool* *]*) – If `True`, a check that the initial_state can fully explore the space will be skipped. (default: `False`) ``` -------------------------------- ### Continuing a Run from HDFBackend Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/monitor.ipynb Restart an MCMC run from a saved state in an HDFBackend file without resetting it. Demonstrates initializing a new sampler with an existing backend and running additional steps. ```python new_backend = emcee.backends.HDFBackend(filename) print("Initial size: {0}".format(new_backend.iteration)) new_sampler = emcee.EnsembleSampler( nwalkers, ndim, log_prob, backend=new_backend ) new_sampler.run_mcmc(None, 100) print("Final size: {0}".format(new_backend.iteration)) ``` -------------------------------- ### Configure Matplotlib Backend and Parameters Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/moves.ipynb Sets up the matplotlib backend for high-resolution figures and adjusts global font and figure sizes. This is boilerplate for plotting. ```python %config InlineBackend.figure_format = "retina" from matplotlib import rcParams rcParams["savefig.dpi"] = 100 rcParams["figure.dpi"] = 100 rcParams["font.size"] = 20 ``` -------------------------------- ### List Keys in HDF5 Backend Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/monitor.ipynb Opens an HDF5 file and prints all top-level keys. Useful for verifying stored runs. ```python import h5py with h5py.File(filename, "r") as f: print(list(f.keys())) ``` -------------------------------- ### KDEMove for Multi-Modal Distributions Source: https://context7.com/dfm/emcee/llms.txt Utilizes emcee's KDEMove for sampling multi-modal distributions by resampling from a kernel density estimate of the ensemble. Best used with many walkers (>=100). Requires emcee, numpy, KDEMove, and scipy. ```python import numpy as np import emcee from emcee.moves import KDEMove # Bimodal 2-D posterior def log_prob(x): log_p1 = -0.5 * np.sum((x - np.array([-3.0, 0.0]))**2) log_p2 = -0.5 * np.sum((x - np.array([ 3.0, 0.0]))**2) return np.logaddexp(log_p1, log_p2) nwalkers, ndim = 100, 2 p0 = np.random.randn(nwalkers, ndim) * 4.0 sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, moves=KDEMove()) sampler.run_mcmc(p0, 1000, progress=False) flat = sampler.get_chain(discard=200, flat=True) print("Samples near mode 1:", np.sum(flat[:, 0] < 0)) print("Samples near mode 2:", np.sum(flat[:, 0] > 0)) ``` -------------------------------- ### Reading Samples from HDFBackend Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/monitor.ipynb Load and analyze MCMC chain properties directly from an HDF5 file using `emcee.backends.HDFBackend`. This allows analysis without needing the original sampler object. ```python reader = emcee.backends.HDFBackend(filename) tau = reader.get_autocorr_time() burnin = int(2 * np.max(tau)) thin = int(0.5 * np.min(tau)) samples = reader.get_chain(discard=burnin, flat=True, thin=thin) log_prob_samples = reader.get_log_prob(discard=burnin, flat=True, thin=thin) log_prior_samples = reader.get_blobs(discard=burnin, flat=True, thin=thin) print("burn-in: {0}".format(burnin)) print("thin: {0}".format(thin)) print("flat chain shape: {0}".format(samples.shape)) print("flat log prob shape: {0}".format(log_prob_samples.shape)) print("flat log prior shape: {0}".format(log_prior_samples.shape)) ``` -------------------------------- ### WalkMove and StretchMove Combination Source: https://context7.com/dfm/emcee/llms.txt Combines emcee's WalkMove with a specified number of helper walkers (s=10) and StretchMove for sampling. Requires emcee, numpy, WalkMove, and StretchMove. ```python import numpy as np import emcee from emcee.moves import WalkMove, StretchMove def log_prob(x): return -0.5 * np.sum(x ** 2) nwalkers, ndim = 50, 6 p0 = np.random.randn(nwalkers, ndim) # WalkMove using 10 helper walkers sampler = emcee.EnsembleSampler( nwalkers, ndim, log_prob, moves=[(WalkMove(s=10), 0.5), (StretchMove(), 0.5)] ) sampler.run_mcmc(p0, 1500, progress=False) print("Acceptance fraction:", np.mean(sampler.acceptance_fraction).round(3)) print("Chain shape:", sampler.get_chain().shape) # (1500, 50, 6) ``` -------------------------------- ### Multiprocessing MCMC with Data Argument Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/parallel.ipynb Illustrates using multiprocessing with emcee where the dataset is passed as an argument. This often shows little to no performance improvement over serial execution due to the overhead of pickling the data for each evaluation. ```python with Pool() as pool: sampler = emcee.EnsembleSampler( nwalkers, ndim, log_prob_data, pool=pool, args=(data,) ) start = time.time() sampler.run_mcmc(initial, nsteps, progress=True) end = time.time() multi_data_time = end - start print("Multiprocessing took {0:.1f} seconds".format(multi_data_time)) print( "{0:.1f} times faster(?) than serial".format( serial_data_time / multi_data_time ) ) ``` -------------------------------- ### Initialize MCMC Run Parameters Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/monitor.ipynb Sets the maximum number of steps for the MCMC run and initializes variables to track the autocorrelation time and convergence. ```python max_n = 100000 # We'll track how the average autocorrelation time estimate changes index = 0 autocorr = np.empty(max_n) # This will be useful to testing convergence old_tau = np.inf ``` -------------------------------- ### emcee.backends.Backend.grow Source: https://github.com/dfm/emcee/blob/main/docs/user/backends.md Expands the storage space for the chain by a specified number of samples. ```APIDOC ## grow ### Description Expand the storage space by some number of samples. ### Method `grow(ngrow, blobs)` ### Parameters - `ngrow` (int): The number of steps to grow the chain. - `blobs`: The current array of blobs. This is used to compute the dtype for the blobs array. ``` -------------------------------- ### emcee Sampler with Blobs for Auxiliary Data Source: https://context7.com/dfm/emcee/llms.txt Shows how to configure an emcee sampler to return auxiliary data (blobs) alongside the log-probability. The log-probability function must return a tuple (log_prob, blob). Blobs are retrieved using get_blobs(). ```python import numpy as np import emcee # Return both the log-posterior and its decomposed components as blobs def log_prob_with_blobs(theta, data): mu, log_sigma = theta if not (-10 < mu < 10 and -5 < log_sigma < 5): return -np.inf, (np.nan, np.nan) sigma = np.exp(log_sigma) log_likelihood = -0.5 * np.sum(((data - mu) / sigma) ** 2) - len(data) * log_sigma log_prior = -0.5 * mu**2 - 0.5 * log_sigma**2 return log_likelihood + log_prior, (log_likelihood, log_prior) data = np.array([1.8, 2.1, 1.95, 2.05, 2.0]) nwalkers, ndim = 20, 2 p0 = np.column_stack([np.random.uniform(1, 3, nwalkers), np.random.uniform(-1, 1, nwalkers)]) sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob_with_blobs, args=[data]) sampler.run_mcmc(p0, 1000, progress=False) # Retrieve blobs: shape (nsteps, nwalkers, 2) blobs = sampler.get_blobs(discard=200, flat=True) print("Blobs shape:", blobs.shape) # (16000, 2) print("Mean log-likelihood:", blobs[:, 0].mean().round(3)) print("Mean log-prior:", blobs[:, 1].mean().round(3)) ``` -------------------------------- ### Read Existing Emcee Chain with HDFBackend Source: https://github.com/dfm/emcee/blob/main/docs/user/backends.md Use HDFBackend to read previously saved sampling results from an HDF5 file. Set read_only=True to prevent accidental overwrites. ```python reader = emcee.backends.HDFBackend("chain_filename.h5", read_only=True) flatchain = reader.get_chain(flat=True) ``` -------------------------------- ### Step MCMC Chain Using Generator Interface Source: https://context7.com/dfm/emcee/llms.txt Uses the `sample` generator to advance the chain step-by-step, allowing for online convergence checks. Demonstrates calculating autocorrelation time and breaking early upon convergence. ```python import numpy as np import emcee def log_prob(x): return -0.5 * np.sum(x ** 2) nwalkers, ndim = 16, 3 p0 = np.random.randn(nwalkers, ndim) sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob) # Use the generator interface for manual control tau_estimates = [] for i, state in enumerate(sampler.sample(p0, iterations=1000, thin_by=10)): if i % 10 == 0 and i > 0: try: tau = sampler.get_autocorr_time(quiet=True) tau_estimates.append(tau.mean()) if tau_estimates[-1] * 50 < sampler.iteration: print(f"Converged at step {sampler.iteration}") break except emcee.autocorr.AutocorrError: pass print("Final state coords shape:", state.coords.shape) # (16, 3) print("Final log_probs:", state.log_prob[:4].round(3)) ``` -------------------------------- ### Optimize Log-Likelihood with SciPy Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/line.ipynb Use scipy.optimize.minimize to find the maximum likelihood estimates for the model parameters (m, b, log_f). Initialize the search near the true values and minimize the negative log-likelihood. ```python from scipy.optimize import minimize np.random.seed(42) nll = lambda *args: -log_likelihood(*args) initial = np.array([m_true, b_true, np.log(f_true)]) + 0.1 * np.random.randn(3) soln = minimize(nll, initial, args=(x, y, yerr)) m_ml, b_ml, log_f_ml = soln.x print("Maximum likelihood estimates:") print("m = {0:.3f}".format(m_ml)) print("b = {0:.3f}".format(b_ml)) print("f = {0:.3f}".format(np.exp(log_f_ml))) ``` -------------------------------- ### emcee Sampler with Named Parameters Source: https://context7.com/dfm/emcee/llms.txt Configures an emcee sampler to pass parameters as a dictionary to the log-probability function by setting `parameter_names`. This is useful for models with semantically meaningful parameter names. ```python import numpy as np import emcee def log_prob(params): # params is now a dict mu = params["mu"] sigma = params["sigma"] if sigma <= 0: return -np.inf data = np.array([2.1, 1.9, 2.3, 2.0, 1.8]) return -0.5 * np.sum(((data - mu) / sigma) ** 2) - len(data) * np.log(sigma) nwalkers, ndim = 20, 2 p0 = np.column_stack([ np.random.uniform(1.5, 2.5, nwalkers), np.random.uniform(0.1, 1.0, nwalkers), ]) sampler = emcee.EnsembleSampler( nwalkers, ndim, log_prob, parameter_names=["mu", "sigma"] ) sampler.run_mcmc(p0, 1000, progress=False) flat = sampler.get_chain(discard=200, flat=True) print("mu =", flat[:, 0].mean().round(3)) print("sigma =", flat[:, 1].mean().round(3)) ``` -------------------------------- ### Write MPI Script Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/parallel.ipynb Writes the emcee MPI execution script to 'script.py'. This script configures and runs an emcee sampler using `MPIPool` for parallel execution. ```python with open("script.py", "w") as f: f.write(""" import sys import time import emcee import numpy as np from schwimmbad import MPIPool def log_prob(theta): t = time.time() + np.random.uniform(0.005, 0.008) while True: if time.time() >= t: break return -0.5*np.sum(theta**2) with MPIPool() as pool: if not pool.is_master(): pool.wait() sys.exit(0) np.random.seed(42) initial = np.random.randn(32, 5) nwalkers, ndim = initial.shape nsteps = 100 sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, pool=pool) start = time.time() sampler.run_mcmc(initial, nsteps) end = time.time() print(end - start) """) ``` -------------------------------- ### Parallel Sampling with multiprocessing.Pool in emcee Source: https://context7.com/dfm/emcee/llms.txt Illustrates how to use `multiprocessing.Pool` with an emcee `EnsembleSampler` for parallel computation of log-probability evaluations. The sampler distributes tasks across the workers automatically. ```python import numpy as np import emcee from multiprocessing import Pool def log_prob(x): # Simulate an expensive likelihood return -0.5 * np.sum(x ** 2) nwalkers, ndim = 32, 8 p0 = np.random.randn(nwalkers, ndim) # The pool object is passed to the sampler with Pool() as pool: sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, pool=pool) sampler.run_mcmc(p0, 1000, progress=True) ``` -------------------------------- ### EnsembleSampler.sample (Generator Interface) Source: https://context7.com/dfm/emcee/llms.txt Demonstrates using the generator interface of `sample()` for step-by-step MCMC execution. This is useful for online convergence checks and adaptive strategies, yielding a `State` object at specified intervals. ```APIDOC ## EnsembleSampler.sample ### Description A generator that advances the chain one step at a time, yielding a `State` object after every `thin_by` proposals. Useful for online convergence checks or adaptive burn-in. ### Method `EnsembleSampler.sample(initial_state, iterations, thin_by=1, **kwargs)` ### Parameters - `initial_state` (State): The starting state of the sampler. - `iterations` (int): The total number of iterations to run. - `thin_by` (int, optional): The number of steps between yielding states. Defaults to 1. - `**kwargs`: Additional keyword arguments forwarded to `sample()`. ### Request Example ```python import numpy as np import emcee def log_prob(x): return -0.5 * np.sum(x ** 2) nwalkers, ndim = 16, 3 p0 = np.random.randn(nwalkers, ndim) sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob) # Use the generator interface for manual control tau_estimates = [] for i, state in enumerate(sampler.sample(p0, iterations=1000, thin_by=10)): if i % 10 == 0 and i > 0: try: tau = sampler.get_autocorr_time(quiet=True) tau_estimates.append(tau.mean()) if tau_estimates[-1] * 50 < sampler.iteration: print(f"Converged at step {sampler.iteration}") break except emcee.autocorr.AutocorrError: pass print("Final state coords shape:", state.coords.shape) # (16, 3) print("Final log_probs:", state.log_prob[:4].round(3)) ``` ### Response #### Success Response Yields `State` objects at intervals defined by `thin_by`. The loop terminates when `iterations` is reached or if an early stopping condition (like convergence) is met. #### Response Example ``` Converged at step 500 Final state coords shape: (16, 3) Final log_probs: [-1.5 -2.1 -0.8 -3.0] ``` ``` -------------------------------- ### emcee.moves.WalkMove Source: https://github.com/dfm/emcee/blob/main/docs/user/moves.md A "walk move" ensemble sampler, based on Goodman & Weare (2010), with parallelization. ```APIDOC ## emcee.moves.WalkMove ### Description A [Goodman & Weare (2010)](https://msp.org/camcos/2010/5-1/p04.xhtml) “walk move” with parallelization as described in [Foreman-Mackey et al. (2013)](https://arxiv.org/abs/1202.3665). ### Parameters * **s** – (optional) The number of helper walkers to use. By default it will use all the walkers in the complement. ``` -------------------------------- ### Plotting Samples Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/quickstart.ipynb Retrieve the flattened chain samples and plot a histogram of the first parameter to visualize the estimated density. ```python import matplotlib.pyplot as plt samples = sampler.get_chain(flat=True) plt.hist(samples[:, 0], 100, color="k", histtype="step") plt.xlabel(r"$\theta_1$") plt.ylabel(r"$p(\theta_1)$") plt.gca().set_yticks([]); ``` -------------------------------- ### Serial MCMC with Data Argument Source: https://github.com/dfm/emcee/blob/main/docs/tutorials/parallel.ipynb Demonstrates a serial MCMC run where a large dataset is passed as an argument to the log probability function. This can lead to performance issues due to repeated pickling. ```python def log_prob_data(theta, data): a = data[0] # Use the data somehow... t = time.time() + np.random.uniform(0.005, 0.008) while True: if time.time() >= t: break return -0.5 * np.sum(theta**2) data = np.random.randn(5000, 200) sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob_data, args=(data,)) start = time.time() sampler.run_mcmc(initial, nsteps, progress=True) end = time.time() serial_data_time = end - start print("Serial took {0:.1f} seconds".format(serial_data_time)) ```