### Development Environment Setup with uv Source: https://github.com/frazane/scoringrules/blob/main/CONTRIBUTING.md Install all project dependencies, including optional backends and development tools, using uv. Run tests for all installed backends or a specific subset. ```bash uv sync --all-extras --dev # all backends (jax/torch are extras) + dev tools uv run pytest tests/ # full suite, every installed backend uv run pytest tests/ --backend numpy # one backend uv run pytest tests/ --backend numpy,numba # a subset uvx pre-commit run --all-files # ruff, codespell, whitespace — the hooks CI runs ``` -------------------------------- ### Install uv and Project Dependencies Source: https://github.com/frazane/scoringrules/blob/main/docs/contributing.md Install the uv package manager and then use it to install project dependencies and development dependencies. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` ```bash uv install ``` -------------------------------- ### Install Pre-commit Hooks Source: https://github.com/frazane/scoringrules/blob/main/docs/contributing.md Install the pre-commit hooks to ensure code quality and consistency before committing changes. ```bash uv run pre-commit install ``` -------------------------------- ### Install scoringrules Source: https://github.com/frazane/scoringrules/blob/main/docs/index.md Install the scoringrules library using pip. Requires Python version 3.11 or higher. ```bash pip install scoringrules ``` -------------------------------- ### Quick Example: CRPS Ensemble Calculation Source: https://github.com/frazane/scoringrules/blob/main/docs/index.md Demonstrates a quick example of calculating the Continuous Ranked Probability Score (CRPS) for an ensemble forecast. This involves generating synthetic observations and forecast data using numpy. ```python import scoringrules as sr import numpy as np obs = np.random.randn(100) fct = obs[:,None] + np.random.randn(100, 21) * 0.1 sr.crps_ensemble(obs, fct) ``` -------------------------------- ### Calculate scoring metrics Source: https://github.com/frazane/scoringrules/blob/main/docs/user_guide.md Examples of calculating various metrics on scalars, arrays, and ensemble data. ```python import numpy as np # on scalars sr.brier_score(0.1, 1) sr.crps_normal(0.1, 1.2, 0.3) # on arrays sr.brier_score(np.random.uniform(0, 1, 100), np.random.binomial(1, 0.5, 100)) sr.crps_lognormal(np.random.lognormal(0, 1, 100), np.random.randn(100), np.random.uniform(0.5, 1.5, 100)) # ensemble metrics obs = np.random.randn(100) fct = obs[:,None] + np.random.randn(100, 21) * 0.1 sr.crps_ensemble(obs, fct) sr.error_spread_score(obs, fct) # multivariate ensemble metrics obs = np.random.randn(100,3) fct = obs[:,None] + np.random.randn(100, 21, 3) * 0.1 sr.energy_score(obs, fct) sr.variogram_score(obs, fct) ``` -------------------------------- ### Serve Documentation Locally Source: https://github.com/frazane/scoringrules/blob/main/docs/contributing.md Build and serve the project's documentation locally using Sphinx and uvx. The documentation will automatically rebuild on file changes. ```bash uvx --with-requirements docs/requirements.txt sphinx-autobuild docs/ docs/_build/ ``` -------------------------------- ### Build and Upload Project Artifacts Source: https://github.com/frazane/scoringrules/blob/main/CONTRIBUTING.md Commands to build the project artifacts and upload them to PyPI. Ensure PyPI upload credentials are configured. ```bash uv build uvx twine check dist/* uvx twine upload dist/* # needs PyPI upload credentials ``` -------------------------------- ### Manage backends Source: https://github.com/frazane/scoringrules/blob/main/docs/user_guide.md Commands to inspect, set, and register computational backends. ```python print(sr.backends) # {'numpy': , # 'numba': } ``` ```python print(sr.backends.active) # ``` ```python sr.backends.set_active("numba") print(sr.backends.active) # ``` ```python sr.register_backend("torch") ``` ```python sr.crps_normal(0.1, 1.0, 0.0, backend="torch") ``` -------------------------------- ### Run Tests with pytest Source: https://github.com/frazane/scoringrules/blob/main/docs/contributing.md Execute the project's test suite using pytest to verify that your changes do not introduce regressions. ```bash uv run pytest tests/ ``` -------------------------------- ### Import scoringrules Source: https://github.com/frazane/scoringrules/blob/main/docs/user_guide.md Initial import required to access the library's functionality. ```python import scoringrules as sr ``` -------------------------------- ### Dry Run Release to TestPyPI Source: https://github.com/frazane/scoringrules/blob/main/CONTRIBUTING.md Perform a dry run of the release process to TestPyPI using the GitHub CLI. This step is crucial before a real release to ensure the workflow functions correctly. ```bash gh workflow run release.yaml -f target=testpypi gh run watch ``` -------------------------------- ### Clone the Repository Source: https://github.com/frazane/scoringrules/blob/main/docs/contributing.md Clone the ScoringRules repository from GitHub to your local machine to begin making changes. ```bash git clone https://github.com//scoringrules.git ``` -------------------------------- ### CRPS Quantile Decomposition (QD) Calculation Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md Calculates the CRPS for an empirical distribution F_M and a verification value y using the quantile decomposition form. This method has O(M*logM) complexity, including sorting. ```python def crps_qd(x, y): """CRPS for empirical distribution F_M and verification y using quantile decomposition. Args: x: Ensemble members (numpy array). y: Verification value. Returns: CRPS value. """ x = np.asarray(x) x.sort() m = len(x) return np.sum((2 / m) * (np.arange(1, m + 1) - 0.5 * (1 + np.arange(1, m + 1) / m)) * (x - y)) ``` -------------------------------- ### Generalised Student's t Distribution PDF+PMF Source: https://github.com/frazane/scoringrules/blob/main/docs/forecast_dists.md Defines the probability density function (pdf) and probability mass function (pmf) for a generalised Student's t distribution. This is used for truncated, censored, or distributions with boundary point masses. ```mathematica f_{\mu, \sigma, \nu, l, L}^{u, U}(x) = \begin{cases} 0, & x < l, \\ L, & x = l, \\ (1 - L - U) f_{\mu, \sigma, \nu}(x), & l < x < u, \\ U, & x = u, \\ 0, & x > u, \end{cases} ``` -------------------------------- ### CRPS Probability Weighted Moment (PWM) Calculation Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md Calculates the CRPS for an empirical distribution F_M and a verification value y using the probability weighted moment form. This method has O(M*logM) complexity, including sorting. ```python def crps_pwm(x, y): """CRPS for empirical distribution F_M and verification y using probability weighted moments. Args: x: Ensemble members (numpy array). y: Verification value. Returns: CRPS value. """ x = np.asarray(x) x.sort() m = len(x) beta_0 = np.mean(x) beta_1 = np.sum((np.arange(m) / m) * x) / (m - 1) return np.mean(np.abs(x - y)) + (m - 1) / m * (beta_0 - 2 * beta_1) ``` -------------------------------- ### Chaining Function for Smooth Gaussian Weight (Diagonal Covariance) Source: https://github.com/frazane/scoringrules/blob/main/docs/weighted_scores.md This formula provides a chaining function for a smooth weight function based on a multivariate Gaussian distribution, assuming a diagonal covariance matrix. It is derived from a component-wise extension of univariate Gaussian chaining functions. ```mathematica v(z) = ((z1 - mu1)Phi(mu1, sigma1)(z1) + sigma1^2 * phi(mu1, sigma1)(z1), ..., (zd - mud)Phi(mud, sigmad)(zd) + sigmad^2 * phi(mud, sigmad)(zd)) ``` -------------------------------- ### Generalised Normal Distribution PDF+PMF Source: https://github.com/frazane/scoringrules/blob/main/docs/forecast_dists.md Defines the probability density function (pdf) and probability mass function (pmf) for a generalised normal distribution. This is used when the distribution is truncated, censored, or has point masses at boundaries. ```mathematica f_{\mu, \sigma, l, L}^{u, U}(x) = \begin{cases} 0, & x < l, \\ L, & x = l, \\ (1 - L - U) f_{\mu, \sigma}(x), & l < x < u, \\ U, & x = u, \\ 0, & x > u, \end{cases} ``` -------------------------------- ### Kernel Score using Conditionally Negative Definite Functions Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md Simplifies the kernel score calculation when the kernel is constructed using a specific form of conditionally negative definite function. This formula is applicable for ensemble forecasts. ```mathematica S_{k}(F_{M}, y) = \frac{1}{M} \sum_{i=1}^{M} \rho(x_{i}, y) - \frac{1}{2 M^{2}} \sum_{i=1}^{M} \sum_{j=1}^{M} \rho(x_{i}, x_{j}). ``` -------------------------------- ### CRPS Energy Form (NRG) Calculation Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md Calculates the CRPS for an empirical distribution F_M and a verification value y using the energy form. This method has O(M^2) complexity. ```python def crps_nrg(x, y): """CRPS for empirical distribution F_M and verification y using energy form. Args: x: Ensemble members (numpy array). y: Verification value. Returns: CRPS value. """ x = np.asarray(x) m = len(x) return np.mean(np.abs(x - y)) - 0.5 * np.mean(np.abs(np.subtract.outer(x, x))) ``` -------------------------------- ### Generalised Student's t Distribution CDF Source: https://github.com/frazane/scoringrules/blob/main/docs/forecast_dists.md Defines the cumulative distribution function (cdf) for a generalised Student's t distribution. It accounts for truncation, censoring, and point masses at the boundaries. ```mathematica F_{\mu, \sigma, \nu, l, L}^{u, U}(x) = \begin{cases} 0, & x < l, \\ L + (1 - L - U) \frac{F_{\mu, \sigma, \nu}(x) - F_{\mu, \sigma, \nu}(l)}{F_{\mu, \sigma, \nu}(u) - F_{\mu, \sigma, \nu}(l)}, & l \le x < u, \\ 1, & x \ge u, \end{cases} ``` -------------------------------- ### Fair Kernel Score for Ensemble Forecasts Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md Provides a fair version of the kernel score for ensemble forecasts by adjusting the empirical means to ensure unbiasedness. This formula is used when the forecast is an ensemble of size M. ```mathematica S_{k}^{f}(F_{M}, y) = \frac{1}{M} \sum_{i=1}^{M} \rho(x_{i}, y) - \frac{1}{2 M(M - 1)} \sum_{i=1}^{M} \sum_{j=1}^{M} \rho(x_{i}, x_{j}). ``` -------------------------------- ### Scoringrules API Overview Source: https://github.com/frazane/scoringrules/blob/main/docs/reference.md This section outlines the main categories of functions available in the scoringrules API. ```APIDOC ## Scoringrules API Reference This page provides a summary of scoringrules’ API. All functions are available in the top-level namespace of the package and are here organized by category. ### Categories - Ensemble forecasts - Univariate - Multivariate - Parametric distributions forecasts - Consistent scoring functions - Categorical forecasts - Backends ``` -------------------------------- ### Chaining Function for Box Weight in Multivariate Space Source: https://github.com/frazane/scoringrules/blob/main/docs/weighted_scores.md This formula defines a chaining function for a multivariate weight function representing a box. It projects points outside the box onto its perimeter while leaving points inside unchanged. ```mathematica v(z) = (min{max{z1, a1}, b1}, ..., min{max{zd, ad}, bd}) ``` -------------------------------- ### Citation for scoringrules Source: https://github.com/frazane/scoringrules/blob/main/docs/index.md Provides the citation details for the scoringrules library in BibTeX format. This should be used when referencing the library in academic work. ```bibtex @software{zanetta_scoringrules_2024, author = {Francesco Zanetta and Sam Allen}, title = {scoringrules: a python library for probabilistic forecast evaluation}, year = {2024}, url = {https://github.com/frazane/scoringrules} } ``` -------------------------------- ### Chaining Function for Multivariate Gaussian Density Source: https://github.com/frazane/scoringrules/blob/main/docs/weighted_scores.md This formula presents a chaining function directly derived from a multivariate Gaussian density weight function. It utilizes the cumulative distribution function of the multivariate Gaussian. ```mathematica v(z) = Phi(mu, Sigma)(z) ``` -------------------------------- ### Fair Kernel Score General Form Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md A general formula for a fair version of the kernel score, applicable when the kernel is positive definite. This formula adjusts the score to be unbiased for ensemble forecasts. ```mathematica S_{k}^{f}(F_{M}, y) = \frac{1}{M(M - 1)} \sum_{i=1}^{M-1} \sum_{j=i+1}^{M} k(x_{i}, x_{j}) + \frac{1}{2} k(y, y) - \frac{1}{M} \sum_{i=1}^{M} k(x_{i}, y). ``` -------------------------------- ### Threshold-Weighted CRPS Formula Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md The threshold-weighted CRPS formula for continuous outcomes, incorporating a weight function. ```mathematica twCRPS(F, y; w) = integral from R of (F(x) - 1{y <= x})^2 w(x) dx = E | v(X) - v(y) | - 1/2 E | v(X) - v(X') | ``` -------------------------------- ### Auxiliary Functions for Generalised t Distribution CRPS Source: https://github.com/frazane/scoringrules/blob/main/docs/forecast_dists.md Defines auxiliary functions G_nu(x), H_nu(x), and B_bar_nu used in the CRPS calculation for the generalised Student's t distribution. These involve the regularised incomplete beta function and the beta function. ```mathematica G_{\nu}(x) &= - \left( \frac{\nu + x^{2}}{\nu - 1} \right) f_{\nu}(x), \\ H_{\nu}(x) &= \frac{1}{2} + \frac{1}{2} \text{sgn}(x) I \left(\frac{1}{2}, \nu - \frac{1}{2}, \frac{x^{2}}{\nu + x^{2}} \right), \\ \bar{B}_{\nu} &= \left( \frac{2 \sqrt{\nu}}{\nu - 1} \right) \frac{B \left(\frac{1}{2}, \nu - \frac{1}{2} \right)}{B(\frac{1}{2}, \frac{\nu}{2})}, \\ z &= \max \{l, \min \{y, u\}\}. ``` -------------------------------- ### Generalised Student's t Distribution CRPS Source: https://github.com/frazane/scoringrules/blob/main/docs/forecast_dists.md Provides the formula for the Continuous Ranked Probability Score (CRPS) of a generalised Student's t distribution. It includes terms for the standard t distribution and boundary point masses. ```mathematica \mathrm{CRPS}(F_{0, 1, \nu, l, L}^{u, U}, y) &= |y - z| + uU^{2} - lL^{2} + \left( \frac{1 - L - U}{F_{\nu}(u) - F_{\nu}(l)} \right) z \left( 2 F_{\nu}(z) - \frac{(1 - 2L)F_{\nu}(u) + (1 - 2U)F_{\nu}(l)}{1 - L - U} \right) - \left( \frac{1 - L - U}{F_{\nu}(u) - F_{\nu}(l)} \right) (2 G_{\nu}(z) - 2 G_{\nu}(u)U - 2 G_{\nu}(l)L) - \left( \frac{1 - L - U}{F_{\nu}(u) - F_{\nu}(l)} \right)^{2} \bar{B}_{\nu} (H_{\nu}(u) - H_{\nu}(l)), \\ \mathrm{CRPS}(F_{\mu, \sigma, \nu, l, L}^{u, U}, y) &= \sigma \mathrm{CRPS} \left( F_{0, 1, \nu, (l - \mu)/\sigma, L}^{(u - \mu)/\sigma, U}, \frac{y - \mu}{\sigma} \right) ``` -------------------------------- ### Quantile Score (Pinball Loss) Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md The quantile score, also known as pinball loss or check loss, is consistent for the alpha-quantile functional. It is used for evaluating quantile forecasts. ```mathematics s_{\alpha}(x, y) = (\mathbf{1}\{y \leq x\} - \alpha)(x - y) = ``` ```mathematics \begin{cases} (1 - \alpha)|x - y| & \quad \text{if } y \leq x \\ \phantom{(1 - }\alpha \phantom{)}\|x - y\| & \quad \text{if } y \geq x \end{cases} ``` -------------------------------- ### Fair CRPS for Ensemble Forecasts Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md The fair version of the Continuous Ranked Probability Score (CRPS) for ensemble forecasts. This formula corrects for potential bias in the standard CRPS calculation when interpreting ensemble members as samples from an underlying distribution. ```mathematica \mathrm{CRPS}^{f}(F_{M}, y) = \frac{1}{M} \sum_{i=1}^{M}|x_i - y| - \frac{1}{2 M (M - 1)}\sum_{i=1}^{M} \sum_{j=1}^{M}|x_i - x_j| ``` -------------------------------- ### Generalised Normal Distribution CDF Source: https://github.com/frazane/scoringrules/blob/main/docs/forecast_dists.md Defines the cumulative distribution function (cdf) for a generalised normal distribution. It accounts for truncation, censoring, and point masses at the boundaries. ```mathematica F_{\mu, \sigma, l, L}^{u, U}(x) = \begin{cases} 0, & x < l, \\ L + (1 - L - U) \frac{F_{\mu, \sigma}(x) - F_{\mu, \sigma}(l)}{F_{\mu, \sigma}(u) - F_{\mu, \sigma}(l)}, & l \le x < u, \\ 1, & x \ge u, \end{cases} ``` -------------------------------- ### Kernel Score for Ensemble Forecasts Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md Calculates the kernel score for ensemble forecasts by replacing expectations with empirical means. This formula is applicable when the forecast is an ensemble of size M. ```mathematica S_{k}(F_{M}, y) = \frac{1}{2} k(y, y) + \frac{1}{2 M^{2}} \sum_{i=1}^{M} \sum_{j=1}^{M} k(x_{i}, x_{j}) - \frac{1}{M} \sum_{i=1}^{M} k(x_{i}, y). ``` -------------------------------- ### Conditional Likelihood Score Formula Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md The conditional likelihood score formula for weighted Log scores, evaluating the conditional distribution given the weight function. ```mathematica coLS(F, y; w) = - w(y) log f(y) + w(y) log(integral from R of w(z) f(z) dz) ``` -------------------------------- ### Log Score for Categorical Outcomes Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md Calculates the Log score for a categorical forecast F and observed outcome y. It focuses on the negative logarithm of the probability assigned to the actual outcome. ```mathematica LS(F, y) = -log F_y ``` -------------------------------- ### Generalised Normal Distribution CRPS Source: https://github.com/frazane/scoringrules/blob/main/docs/forecast_dists.md Provides the formula for the Continuous Ranked Probability Score (CRPS) of a generalised normal distribution. It includes terms for the standard normal distribution and boundary point masses. ```mathematica \mathrm{CRPS}(F_{0, 1, l, L}^{u, U}, y) &= |y - z| + uU^{2} - lL^{2} + \left( \frac{1 - L - U}{\Phi(u) - \Phi(l)} \right) z \left( 2 \Phi(z) - \frac{(1 - 2L)\Phi(u) + (1 - 2U)\Phi(l)}{1 - L - U} \right) + \left( \frac{1 - L - U}{\Phi(u) - \Phi(l)} \right) (2 \phi(z) - 2 \phi(u)U - 2 \phi(l)L) - \left( \frac{1 - L - U}{\Phi(u) - \Phi(l)} \right)^{2} \left( \frac{1}{\sqrt{\pi}} \right) \left(\Phi \left( u\sqrt{2} \right) - \Phi \left( l\sqrt{2} \right) \right), \mathrm{CRPS}(F_{\mu, \sigma, l, L}^{u, U}, y) &= \sigma \mathrm{CRPS} \left( F_{0, 1, (l - \mu)/\sigma, L}^{(u - \mu)/\sigma, U}, \frac{y - \mu}{\sigma} \right) ``` -------------------------------- ### Ranked Probability Score (RPS) Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md Calculates the Ranked Probability Score for an ordinal categorical forecast F and outcome y. It uses cumulative probabilities and indicator functions to account for category ordering. ```mathematica RPS(F, y) = sum_{i=1}^{K} (tilde{F}_j - tilde{y}_j)^2 ``` -------------------------------- ### CRPS for Ensemble Forecasts Source: https://github.com/frazane/scoringrules/blob/main/docs/crps_estimators.md The Continuous Ranked Probability Score (CRPS) formula for ensemble forecasts, derived from the kernel score definition with rho(x, x') = |x - x'|. This is the standard, potentially biased, version. ```mathematica \mathrm{CRPS}(F_{M}, y) = \frac{1}{M} \sum_{i=1}^{M}|x_i - y| - \frac{1}{2 M^2}\sum_{i=1}^{M} \sum_{j=1}^{M}|x_i - x_j| ``` -------------------------------- ### Censored Likelihood Score Formula Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md The censored likelihood score formula for weighted Log scores, as proposed by Diks et al. (2011). ```mathematica ceLS(F, y; w) = - w(y) log f(y) - (1 - w(y)) log(1 - integral from R of w(z) f(z) dz) ``` -------------------------------- ### Interval Score for Prediction Intervals Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md The interval score evaluates prediction intervals defined by a lower (L) and upper (U) bound. It penalizes intervals that do not contain the observed value and considers the width of the interval. ```mathematics \mathrm{IS}([L, U], y) = |U - L| + \frac{2}{\alpha} (y - u) \mathbf{1} \{ y > u \} + \frac{2}{\alpha} (l - y) \mathbf{1} \{ y < l \}. ``` -------------------------------- ### Brier Score for Categorical Outcomes Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md Calculates the Brier score for a categorical forecast F and observed outcome y. It sums the squared differences between forecast probabilities and indicator functions for each category. ```mathematica BS_K(F, y) = sum_{i=1}^{K} (F_i - 1{y = i})^2 ``` -------------------------------- ### Ranked Logarithmic Score (RLS) Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md Calculates the Ranked Logarithmic Score for an ordinal categorical forecast F and outcome y. It uses cumulative probabilities and the absolute difference of cumulative terms. ```mathematica RLS(F, y) = - sum_{i=1}^{K} log | tilde{F}_j + tilde{y}_j - 1| ``` -------------------------------- ### Absolute Error Scoring Function Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md The absolute error scoring function is consistent for the median functional. It is a special case of the quantile score when alpha is 0.5. ```mathematics s(x, y) = |x - y| ``` -------------------------------- ### Squared Error Scoring Function Source: https://github.com/frazane/scoringrules/blob/main/docs/theory.md The squared error scoring function is consistent for the mean functional. It is used to evaluate forecasts where the mean is the primary interest. ```mathematics s(x, y) = (x - y)^{2} ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.