### Install multiversum package Source: https://simson.io/multiversum Installs the multiversum library using pip. ```bash pip install multiversum ``` -------------------------------- ### get_run_dir(sub_directory=None) Source: https://simson.io/multiversum/api/multiverse_analysis Gets the directory for the current run, optionally appending a sub-directory. Creates the directory if it does not exist. ```APIDOC ## get_run_dir(sub_directory=None) ### Description Get the directory for the current run. Optionally appends a sub-directory to the run directory and creates the directory if it does not exist. ### Method Method call (Python) ### Endpoint `get_run_dir(sub_directory=None)` ### Parameters #### Path Parameters - **sub_directory** (Optional[str]) - Optional - An optional sub-directory to append to the run directory. Default is `None`. ### Returns - **Path** - A Path object for the current run directory. ### Source Code ```python def get_run_dir(self, sub_directory: Optional[str] = None) -> Path: """ Get the directory for the current run. Args: sub_directory: An optional sub-directory to append to the run directory. Returns: A Path object for the current run directory. """ run_dir = self.output_dir / "runs" / str(self.run_no) target_dir = run_dir / sub_directory if sub_directory is not None else run_dir target_dir.mkdir(parents=True, exist_ok=True) return target_dir ``` ``` -------------------------------- ### Universe.__init__ Source: https://simson.io/multiversum/api/universe Initializes a Universe analysis instance. It parses the settings dictionary, handles seed overriding, and sets up dimensions, output directory, and metrics. ```APIDOC ## Universe.__init__ ### Description Initializes a Universe analysis instance. It parses the settings dictionary, handles seed overriding, and sets up dimensions, output directory, and metrics. ### Method Constructor ### Endpoint Universe.__init__ ### Parameters #### Constructor Parameters - **settings** (dict or str) - Required - The settings for the universe analysis. This can usually just be passed along from the multiverse analysis. You only need to specify this yourself when developing / trying out an analysis. Possible keys in the dictionary are: - dimensions: The specified universe dimensions. This is the only required information. - run_no: The run number of the multiverse analysis. - seed: The seed to use for analyses. - universe_id: The id of the universe. - output_dir: The directory to which the output should be written. - **metrics** (dict) - Required - A dictionary containing the metrics to be computed. Pass an empty dictionary to not compute any. - **fairness_metrics** (dict) - Required - A dictionary containing the fairness metrics to be computed. (These are computed with awareness of groups.) Pass an empty dictionary to not compute any. - **expand_dicts** (bool) - Optional - Whether to expand dictionaries in the dimensions i.e. if there are any dictionaries in the dimensions, expand them into separate dimensions of their own. Defaults to False. - **seed** (int or str) - Optional - An optional seed value to override the one from settings. This can be either: - an integer: directly used as seed value - a string: interpreted as a dimension name whose value will be used as seed - **set_seed** (bool) - Optional - Whether to use the seed provided in the settings. Defaults to True. Please note, that this only sets the seed in the Python random module and numpy. ### Request Example ```python Universe( settings={ "dimensions": {"age": 30, "gender": "female"}, "run_no": 1, "seed": 42, "universe_id": "universe-1", "output_dir": "./output" }, metrics={"accuracy": "accuracy_score"}, fairness_metrics={"equalized_odds": "equalized_odds"}, expand_dicts=False, seed=None, set_seed=True ) ``` ### Response #### Success Response No return value. The constructor initializes the Universe instance with the provided settings. #### Response Example No response body. ``` -------------------------------- ### get_execution_time() Source: https://simson.io/multiversum/api/universe Gets the execution time of the universe analysis. Returns the elapsed time in seconds as a float. ```APIDOC ## get_execution_time() ### Description Gets the execution time of the universe analysis. ### Method Function call ### Endpoint `get_execution_time()` ### Parameters None ### Returns - **float** (float) - The execution time in seconds. ### Source Code ```python def get_execution_time(self) -> float: """ Gets the execution time of the universe analysis. Returns: float: The execution time in seconds. """ if self.ts_end is None: print("Stopping execution_time clock.") self.ts_end = time.time() return self.ts_end - self.ts_start ``` ``` -------------------------------- ### MultiverseAnalysis __init__ constructor Source: https://simson.io/multiversum/api/multiverse_analysis Initializes a MultiverseAnalysis instance. Accepts dimensions, config (TOML/JSON/Python), universe file, output directory, run number, and other options. Searches for config files automatically if not provided, parses them, and converts to a Config object. Raises ValueError if dimensions are not specified. ```python def __init__( self, dimensions: Optional[Dict] = None, config: Union[Path, Config, None] = None, universe: Path = None, output_dir: Path = Path("./output"), run_no: Optional[int] = None, new_run: bool = True, seed: Optional[int] = None, stop_on_error: Optional[bool] = None, cell_timeout: Optional[int] = None, ) -> None: """ Initializes a new MultiverseAnalysis instance. Args: dimensions: A dictionary where keys are dimension names and values are lists of possible values for each dimension. Each dimension corresponds to a decision. config: A Path to a TOML, JSON or Python file containing the analysis configuration. Supported configuration options can be found in the Config class. If a Python file is used, it should contain a dictionary / config object named "config". Will automatically search for multiverse.toml / .json / .py. universe: The Path to the universe_file to run. Either an ipython / jupyter notebook (.ipynb) or a python script (.py). output_dir: The directory to store the output in. run_no: The number of the current run. Defaults to an automatically incrementing integer number if new_run is True or the last run if new_run is False. new_run: Whether this is a new run or not. Defaults to True. seed: The seed to use for the analysis. stop_on_error: Whether to stop the analysis if an error occurs. cell_timeout: A timeout (in seconds) for each cell in the notebook. """ # Check for configuration file and parse it config_file = search_files(file=config, default_files=DEFAULT_CONFIG_FILES) if isinstance(config_file, Path): if config_file.suffix == ".toml": with open(config_file, "rb") as fp: config = tomllib.load(fp) elif config_file.suffix == ".json": with open(config_file, "r") as fp: config = json.load(fp) elif config_file.suffix == ".py": results = runpy.run_path(str(config_file)) config = results["config"] else: raise ValueError( "Only .toml, .json and .py files are supported as config." ) # Convert config to Config object if isinstance(config, dict): config = Config(**config) # Read settings from config (or args) self.read_config_value(config, "dimensions", dimensions) self.read_config_value(config, "seed", seed) self.read_config_value(config, "stop_on_error", stop_on_error) self.read_config_value(config, "cell_timeout", cell_timeout) self.read_config_value(config, "constraints") universe_file = search_files( file=universe, default_files=DEFAULT_UNIVERSE_FILES ) self.universe_file = universe_file self.output_dir = output_dir if self.output_dir is not None: self.output_dir.mkdir(parents=True, exist_ok=True) self.run_no = ( run_no if run_no is not None else self.read_counter(increment=new_run) ) if self.dimensions is None: raise ValueError( "Dimensions need to be specified either directly or in a config." ) ``` -------------------------------- ### MultiverseAnalysis.__init__ Source: https://simson.io/multiversum/api/multiverse_analysis Initializes a new MultiverseAnalysis instance. Accepts dimensions, configuration, universe file, and various run settings to set up the analysis. ```APIDOC ## MultiverseAnalysis.__init__ ### Description Initializes a new MultiverseAnalysis instance with the provided dimensions, configuration, universe file, and run settings. ### Method Constructor ### Signature ```python def __init__( self, dimensions: Optional[Dict] = None, config: Union[Path, Config, None] = None, universe: Path = None, output_dir: Path = Path("./output"), run_no: Optional[int] = None, new_run: bool = True, seed: Optional[int] = None, stop_on_error: Optional[bool] = None, cell_timeout: Optional[int] = None, ) -> None ``` ### Parameters - **dimensions** (Optional[Dict]) - Optional - A dictionary where keys are dimension names and values are lists of possible values for each dimension. Each dimension corresponds to a decision. - **config** (Union[Path, Config, None]) - Optional - A Path to a TOML, JSON or Python file containing the analysis configuration. Supported configuration options can be found in the Config class. If a Python file is used, it should contain a dictionary / config object named "config". Will automatically search for multiverse.toml / .json / .py. - **universe** (Path) - Optional - The Path to the universe_file to run. Either an ipython / jupyter notebook (.ipynb) or a python script (.py). - **output_dir** (Path) - Optional - The directory to store the output in. Defaults to Path("./output"). - **run_no** (Optional[int]) - Optional - The number of the current run. Defaults to an automatically incrementing integer number if new_run is True or the last run if new_run is False. - **new_run** (bool) - Optional - Whether this is a new run or not. Defaults to True. - **seed** (Optional[int]) - Optional - The seed to use for the analysis. - **stop_on_error** (Optional[bool]) - Optional - Whether to stop the analysis if an error occurs. - **cell_timeout** (Optional[int]) - Optional - A timeout (in seconds) for each cell in the notebook. ### Description of Behavior - Checks for a configuration file and parses it if provided. Supported formats: .toml, .json, .py. Raises ValueError for unsupported formats. - Converts config to a Config object if a dictionary is provided. - Reads settings from config or arguments for dimensions, seed, stop_on_error, cell_timeout, and constraints. - Searches for the universe file using default universe files if not provided. - Creates the output directory if it does not exist. - Determines run_no based on run_no argument and new_run flag. - Raises ValueError if dimensions are not specified either directly or in config. ### Raises - **ValueError** - If the config file suffix is not .toml, .json, or .py. - **ValueError** - If dimensions are not specified either directly or in config. ``` -------------------------------- ### get_export_file_path(filename, mkdir=False) Source: https://simson.io/multiversum/api/universe Get the file path for exporting a file from this Universe. Returns a Path object representing the full export path. ```APIDOC ## get_export_file_path(filename, mkdir=False) ### Description Get the file path for exporting a file from this Universe. ### Method Function call ### Endpoint `get_export_file_path(filename, mkdir=False)` ### Parameters #### Path Parameters - **filename** (str) - Required - The name of the file to export. - **mkdir** (bool) - Optional - Whether to create the directory if it does not exist. Default: `False` ### Returns - **Path** (Path) - The full path where the file should be exported. ### Source Code ```python def get_export_file_path(self, filename: str, mkdir: bool = False) -> Path: """ Get the file path for exporting a file from this Universe. Args: filename: The name of the file to export. mkdir: Whether to create the directory if it does not exist. (optional) Returns: Path: The full path where the file should be exported. """ export_dir = ( self.output_dir / "runs" / str(self.run_no) / "exports" / self.universe_id ) if mkdir: export_dir.mkdir(parents=True, exist_ok=True) return export_dir / filename ``` ``` -------------------------------- ### MultiverseAnalysis.__init__ Source: https://simson.io/multiversum/api/multiverse_analysis Initializes a new MultiverseAnalysis instance with the provided dimensions, configuration, universe file, and execution settings. ```APIDOC ## MultiverseAnalysis.__init__ ### Description Initializes a new MultiverseAnalysis instance. Accepts dimensions, an optional configuration file (TOML, JSON, or Python), a universe file (notebook or script), output directory, run number, and execution settings. ### Method Constructor ### Signature ```python MultiverseAnalysis( dimensions: Optional[Dict] = None, config: Union[Path, Config, None] = None, universe: Path = None, output_dir: Path = Path("./output"), run_no: Optional[int] = None, new_run: bool = True, seed: Optional[int] = None, stop_on_error: Optional[bool] = None, cell_timeout: Optional[int] = None, ) -> None ``` ### Parameters - **dimensions** (Optional[Dict]) - Optional - A dictionary where keys are dimension names and values are lists of possible values for each dimension. Each dimension corresponds to a decision. - **config** (Union[Path, Config, None]) - Optional - A Path to a TOML, JSON or Python file containing the analysis configuration. Supported configuration options can be found in the Config class. If a Python file is used, it should contain a dictionary / config object named "config". Will automatically search for multiverse.toml / .json / .py. - **universe** (Path) - Optional - The Path to the universe_file to run. Either an ipython / jupyter notebook (.ipynb) or a python script (.py). - **output_dir** (Path) - Optional - The directory to store the output in. Defaults to Path("./output"). - **run_no** (Optional[int]) - Optional - The number of the current run. Defaults to an automatically incrementing integer number if new_run is True or the last run if new_run is False. - **new_run** (bool) - Optional - Whether this is a new run or not. Defaults to True. - **seed** (Optional[int]) - Optional - The seed to use for the analysis. - **stop_on_error** (Optional[bool]) - Optional - Whether to stop the analysis if an error occurs. - **cell_timeout** (Optional[int]) - Optional - A timeout (in seconds) for each cell in the notebook. ### Configuration File Support - **.toml** - Parsed using tomllib. - **.json** - Parsed using json.load. - **.py** - Executed with runpy, expecting a variable named "config". - Other extensions raise a ValueError. ### Notes - The constructor reads settings from the config file (or arguments) for dimensions, seed, stop_on_error, cell_timeout, and constraints. - The `constraints` attribute is an optional dictionary where keys are dimension names and values are lists of constraints. Each constraint is a dictionary with: - **value** - The value of the dimension that the constraint applies to. - **allowed_if** - A dictionary of dimension-value pairs that must be present for the constraint to be allowed. - **forbidden_if** - A dictionary of dimension-value pairs that must not be present for the constraint to be allowed. - Only one of allowed_if and forbidden_if can be present in a constraint. ``` -------------------------------- ### read_counter(increment) method implementation Source: https://simson.io/multiversum/api/multiverse_analysis Reads the counter from counter.txt in the output directory. If the file does not exist, starts at 0. If increment is True, increments the counter before writing it back. Returns the current (or incremented) value. ```python def read_counter(self, increment: bool) -> int: """ Read the counter from the output directory. Args: increment: Whether to increment the counter after reading. Returns: The current value of the counter. """ # Use a self-incrementing counter via counter.txt counter_filepath = self.output_dir / "counter.txt" if counter_filepath.is_file(): with open(counter_filepath, "r") as fp: run_no = int(fp.read()) else: run_no = 0 if increment: run_no += 1 with open(counter_filepath, "w") as fp: fp.write(str(run_no)) return run_no ``` -------------------------------- ### Universe.__init__ Source: https://simson.io/multiversum/api/universe Initializes the Universe class with settings, optional metrics and fairness metrics, and options for expanding dictionaries and setting a seed. Arguments should be passed in from the larger multiverse analysis. ```APIDOC ## Universe.__init__ ### Description Initializes the Universe class with settings, optional metrics and fairness metrics, and options for expanding dictionaries and setting a seed. Arguments should be passed in from the larger multiverse analysis. ### Method Constructor ### Signature ```python def __init__( self, settings: Union[str, Dict[str, Any]], metrics: Optional[Dict[str, Callable]] = None, fairness_metrics: Optional[Dict[str, Callable]] = None, expand_dicts: bool = False, seed: Optional[Union[int, str]] = None, set_seed: bool = True, ) -> None ``` ### Parameters - **settings** (Union[str, Dict[str, Any]]) - Required - The settings for the universe. - **metrics** (Optional[Dict[str, Callable]]) - Optional - A dictionary containing the metrics to be computed. - **fairness_metrics** (Optional[Dict[str, Callable]]) - Optional - A dictionary containing the fairness metrics to be computed. - **expand_dicts** (bool) - Optional - Whether to expand dictionaries in the settings. - **seed** (Optional[Union[int, str]]) - Optional - The seed to use for reproducibility. - **set_seed** (bool) - Optional - Whether to set the seed. ### Returns None ``` -------------------------------- ### Run multiverse analysis with uv Source: https://simson.io/multiversum Runs the multiverse analysis using the uv package manager. ```bash uv run -m multiversum ``` -------------------------------- ### MultiverseAnalysis.__init__ Source: https://simson.io/multiversum/api/multiverse_analysis Initializes a new MultiverseAnalysis instance with configurable dimensions, configuration, universe file, output directory, run number, seed, error handling, and cell timeout. ```APIDOC ## MultiverseAnalysis.__init__ ### Description Initializes a new MultiverseAnalysis instance. ### Method Constructor ### Signature `__init__(dimensions=None, config=None, universe=None, output_dir=Path('./output'), run_no=None, new_run=True, seed=None, stop_on_error=None, cell_timeout=None)` ### Parameters - **dimensions** (Optional[Dict]) - Optional - A dictionary where keys are dimension names and values are lists of possible values for each dimension. Each dimension corresponds to a decision. - **config** (Union[Path, Config, None]) - Optional - A Path to a TOML, JSON or Python file containing the analysis configuration. Supported configuration options can be found in the Config class. If a Python file is used, it should contain a dictionary / config object named "config". Will automatically search for multiverse.toml / .json / .py. - **universe** (Path) - Optional - The Path to the universe_file to run. Either an ipython / jupyter notebook (.ipynb) or a python script (.py). - **output_dir** (Path) - Optional - The directory to store the output in. Defaults to Path('./output'). - **run_no** (Optional[int]) - Optional - The number of the current run. Defaults to an automatically incrementing integer number if new_run is True or the last run if new_run is False. - **new_run** (bool) - Optional - Whether this is a new run or not. Defaults to True. - **seed** (Optional[int]) - Optional - The seed to use for the analysis. - **stop_on_error** (Optional[bool]) - Optional - Whether to stop the analysis if an error occurs. - **cell_timeout** (Optional[int]) - Optional - A timeout (in seconds) for each cell in the notebook. ### Source Code Source code in `multiversum/multiverse.py` ``` -------------------------------- ### Universe class initialization and settings parsing Source: https://simson.io/multiversum/api/universe Initializes the Universe object by parsing settings, expanding dimensions, handling seed overrides, and setting up output directory and metrics. Warns when seed is overridden or dimensions are missing. ```python # Extract settings parsed_settings = ( json.loads(settings) if isinstance(settings, str) else settings ) self.run_no = parsed_settings["run_no"] if "run_no" in parsed_settings else 0 self.universe_id = ( parsed_settings["universe_id"] if "universe_id" in parsed_settings else "no-universe-id-provided" ) self.dimensions = parsed_settings["dimensions"] if expand_dicts: # Create a new dictionary to store expanded dimensions expanded_dimensions = {} # Process each key-value pair in dimensions for key, value in self.dimensions.items(): if isinstance(value, dict): # For dictionary values, add all key-value pairs to root level # but don't expand nested dictionaries further expanded_dimensions.update(value) else: # For non-dictionary values, keep them as is expanded_dimensions[key] = value self.dimensions = expanded_dimensions # Handle seed overriding settings_seed = parsed_settings["seed"] if "seed" in parsed_settings else 0 if seed is not None: # If seed is a string, try to interpret it as a dimension name if isinstance(seed, str): if seed in self.dimensions: seed_value = self.dimensions[seed] if "seed" in parsed_settings: warnings.warn( f"Seed from dimension '{seed}' ({seed_value}) is overriding seed from settings ({settings_seed})." ) self.seed = seed_value else: warnings.warn( f"Dimension '{seed}' not found in dimensions. Using settings seed ({settings_seed}) instead." ) self.seed = settings_seed else: # Seed is a direct value if "seed" in parsed_settings: warnings.warn( f"Seed provided in constructor ({seed}) is overriding seed from settings ({settings_seed})." ) self.seed = seed else: self.seed = settings_seed self.output_dir = ( Path(parsed_settings["output_dir"]) if "output_dir" in parsed_settings else Path("./output") ) self.metrics = metrics self.fairness_metrics = fairness_metrics if self.dimensions is None: warnings.warn("No dimensions specified for universe analysis.") if set_seed: print(f"Setting seed to {self.seed} (in: [random, numpy.random]).") random.seed(self.seed) np.random.seed(self.seed) ``` -------------------------------- ### Universe.__init__ Source: https://simson.io/multiversum/api/universe Initializes the Universe class. The arguments should be passed in from the larger multiverse analysis. ```APIDOC ## Universe.__init__ ### Description Initializes the Universe class. The arguments should be passed in from the larger multiverse analysis. ### Method Constructor ### Signature ```python def __init__( self, settings: Union[str, Dict[str, Any]], metrics: Optional[Dict[str, Callable]] = None, fairness_metrics: Optional[Dict[str, Callable]] = None, expand_dicts: bool = False, seed: Optional[Union[int, str]] = None, set_seed: bool = True, ) -> None: ``` ### Parameters - **settings** (Union[str, Dict[str, Any]]) - Required - The settings for the universe analysis. This can usually just be passed along from the multiverse analysis. You only need to specify this yourself when developing / trying out an analysis. Possible keys in the dictionary are: dimensions (required), run_no, seed, universe_id, output_dir. - **metrics** (Optional[Dict[str, Callable]]) - Optional - A dictionary containing the metrics to be computed. Pass an empty dictionary to not compute any. - **fairness_metrics** (Optional[Dict[str, Callable]]) - Optional - A dictionary containing the fairness metrics to be computed. (These are cumputed with awareness of groups.) Pass an empty dictionary to not compute any. - **expand_dicts** (bool) - Optional - Whether to expand dictionaries in the dimensions i.e. if there are any dictionaries in the dimensions, expand them into separate dimensions of their own. Defaults to False. - **seed** (Optional[Union[int, str]]) - Optional - An optional seed value to override the one from settings. This can be either: an integer: directly used as seed value; a string: interpreted as a dimension name whose value will be used as seed. - **set_seed** (bool) - Optional - Whether to use the seed provided in the settings. Defaults to True. Please note, that this only sets the seed in the Python random module and numpy. ### Returns None ``` -------------------------------- ### Distributed execution with --mode=partial-parallel and --mode=finalize Source: https://simson.io/multiversum Run a distributed multiverse analysis across multiple machines using partial-parallel mode. Use --partial with percentage ranges on each machine (e.g., 0%,50% and 50%,100%), then run --mode=finalize after all machines complete. ```shell # On machine 1: Run first 33% of universes python -m multiversum --mode=partial-parallel --partial="0%,50%" # On machine 2: Run second 33% of universes python -m multiversum --mode=partial-parallel --partial="50%,100%" # After all machines complete, finalize the analysis python -m multiversum --mode=finalize ``` -------------------------------- ### execute_notebook_via_api(input_path, output_path, parameters) Source: https://simson.io/multiversum/api/multiverse_analysis Executes a notebook via the papermill Python API. This method takes an input notebook path, an output notebook path, and a dictionary of parameters, and runs the notebook with those parameters. ```APIDOC ## execute_notebook_via_api(input_path, output_path, parameters) ### Description Executes a notebook via the papermill Python API. ### Method Python method ### Endpoint `execute_notebook_via_api(input_path, output_path, parameters)` ### Parameters - **input_path** (str) - Required - The path to the input notebook. - **output_path** (str) - Required - The path to the output notebook. - **parameters** (Dict[str, str]) - Required - A dictionary containing the parameters for the notebook. ### Returns - **None** - None ### Source Code ```python def execute_notebook_via_api( self, input_path: str, output_path: str, parameters: Dict[str, str] ) -> None: """ Executes a notebook via the papermill python API. Args: input_path: The path to the input notebook. output_path: The path to the output notebook. parameters: A dictionary containing the parameters for the notebook. Returns: None """ pm.execute_notebook( input_path, output_path, parameters=parameters, progress_bar=False, kernel_manager_class="multiversum.IPCKernelManager.IPCKernelManager", execution_timeout=self.cell_timeout, ) ``` ``` -------------------------------- ### python -m multiversum run options Source: https://simson.io/multiversum/cli Shows the options for the run subcommand. ```bash --mode [full|continue|test|partial-parallel|finalize] How to run the multiverse analysis.(continue: continue from previous run, full: run all universes, test: run a minimal set of universes where each unique option appears at least once, partial- parallel: run a partial range of universes in parallel see --partialfinalize: finalize the previous run e.g. after running in partial-parallel mode) --config PATH Relative path to a TOML, JSON or Python file with a config for the multiverse. Defaults to searching for multiverse.toml, multiverse.json, multiverse.py (in that order). --universe PATH Relative path to the universe file to run. Defaults to searching for universe.ipynb, universe.py (in that order). --output-dir PATH Relative path to output directory for the results. --seed INTEGER The seed to use for the analysis (Defaults to 80539). --u-id TEXT Examine only a single universe with the given universe id (or starting with the provided id). --grid-only Only export the multiverse grid without running the analysis. --grid-format [json|csv|none] Format of the exported multiverse grid (json, csv, or none to skip export). --n-jobs, --njobs INTEGER Number of CPUs to use for parallel processing. -1 uses all CPUs, -2 uses all but one CPU (default), and 1 disables parallel processing. --partial TEXT Run only a specific percentage range of universes. Format: 'start%,end%' (e.g. '0%,50%' or '0,20%'). Set mode to 'partial- parallel' to run in parallel and avoid race conditions. --retry-errors When mode=continue, retry universes that previously resulted in errors. --retry-error-type TEXT When mode=continue, retry only universes that failed with this specific error type. --help Show this message and exit. ``` -------------------------------- ### Implement analysis in universe.py with multiversum and scikit-learn Source: https://simson.io/multiversum Implements the actual analysis in a Python file ('universe.py') using the multiversum Universe class and scikit-learn classifiers. It demonstrates how to access dimension settings dynamically and save results. ```python # Implement the actual analysis import pandas as pd from sklearn.linear_model import LogisticRegression from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from multiversum import Universe # Initialize the Universe with example settings for development # These will automatically be overwritten by multiversum in the analysis universe = Universe(settings = { # Automatically overwritten "classifier": "logistic", "feature_set": "standard", }) seed = universe.seed # Load the data df = pd.read_csv("https://raw.githubusercontent.com/allisonhorst/palmerpenguins/main/inst/extdata/penguins.csv") # Prepare data based on feature_set dimension X = None if universe.dimensions["feature_set"] == "minimal": X = df[["flipper_length_mm"]] elif universe.dimensions["feature_set"] == "standard": X = df[["flipper_length_mm", "bill_length_mm"]] elif universe.dimensions["feature_set"] == "full": X = df[["flipper_length_mm", "bill_length_mm", "bill_depth_mm"]] y = df["species"] # Split data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed) # Choose classifier based on dimension model = None if universe.dimensions["classifier"] == "logistic": model = LogisticRegression(random_state=seed) elif universe.dimensions["classifier"] == "random_forest": model = RandomForestClassifier(random_state=seed) # Train model model.fit(X_train, y_train) # Evaluate y_pred = model.predict(X_test) accuracy = accuracy_score(y_test, y_pred) # Prepare results = pd.DataFrame({ "accuracy": [accuracy], "feature_count": [len(X.columns)] }) # Let multiversum handle the storage / saving universe.save_data(results) ``` -------------------------------- ### get_run_dir(sub_directory=None) method implementation Source: https://simson.io/multiversum/api/multiverse_analysis Returns a Path object for the current run directory, creating it if necessary. If sub_directory is provided, it is appended to the run directory. The run directory is constructed as output_dir/runs/run_no. ```python def get_run_dir(self, sub_directory: Optional[str] = None) -> Path: """ Get the directory for the current run. Args: sub_directory: An optional sub-directory to append to the run directory. Returns: A Path object for the current run directory. """ run_dir = self.output_dir / "runs" / str(self.run_no) target_dir = run_dir / sub_directory if sub_directory is not None else run_dir target_dir.mkdir(parents=True, exist_ok=True) return target_dir ``` -------------------------------- ### python -m multiversum --help output Source: https://simson.io/multiversum/cli Shows the help output for the main multiversum CLI command. ```bash --help Show this message and exit. ``` -------------------------------- ### Universe class definition and initializer Source: https://simson.io/multiversum/api/universe Defines the core Universe class with class-level defaults for metrics, fairness metrics, and timestamps, plus the __init__ method that accepts universe settings from the multiverse configuration. The settings parameter is typed as either a string or a dictionary. ```python class Universe: """ A class to help with running the analysis of a single universe contained within a multiverse analysis. Attributes: run_no: The run number of the multiverse analysis. universe_id: The id of the universe. universe: The universe settings. output_dir: The directory to which the output should be written. metrics: A dictionary containing the metrics to be computed. fairness_metrics: A dictionary containing the fairness metrics to be computed. ts_start: The timestamp of the start of the analysis. ts_end: The timestamp of the end of the analysis. """ metrics = None fairness_metrics = None ts_start = None ts_end = None def __init__( self, settings: Union[str, Dict[str, Any]], metrics: Optional[Dict[str, Callable]] = None, fairness_metrics: Optional[Dict[str, Callable]] = None, expand_dicts: bool = False, seed: Optional[Union[int, str]] = None, set_seed: bool = True, ) -> None: """ Initialize the Universe class. The arguments should be passed in from the larger multiverse analysis. """ ``` -------------------------------- ### visit_universe method implementation in multiversum/multiverse.py Source: https://simson.io/multiversum/api/multiverse_analysis Runs the complete analysis for a single universe. Generates a universe ID, removes old error files, builds the output path, and executes either a .ipynb notebook via API or a .py script. On error, renames the output file to indicate an error and either raises or logs and saves the error depending on stop_on_error. ```python def visit_universe(self, universe_dimensions: Dict[str, str]) -> None: """ Run the complete analysis for a single universe. Output from the analysis will be saved to a file in the run's output directory. Args: universe_dimensions: A dictionary containing the parameters for the universe. Returns: None """ # Generate universe ID universe_id = generate_universe_id(universe_dimensions) logger.debug(f"Visiting universe: {universe_id}") # Clean up any old error fiels error_path = self._get_error_filepath(universe_id) if error_path.is_file(): logger.warning( f"Removing old error file: {error_path}. This should only happen during a re-run." ) error_path.unlink() universe_filetype = self.universe_file.suffix # Generate final command output_dir = self.get_run_dir(sub_directory="universes") output_filename = f"nb_{self.run_no}-{universe_id}{universe_filetype}" output_path = output_dir / output_filename # Ensure output dir exists output_dir.mkdir(parents=True, exist_ok=True) # Prepare settings dictionary settings = { "universe_id": universe_id, "dimensions": universe_dimensions, "run_no": self.run_no, "output_dir": str(self.output_dir), "seed": self.seed, } settings_str = json.dumps(settings, sort_keys=True) try: with warnings.catch_warnings(record=True) as w: warnings.simplefilter("always") if universe_filetype == ".ipynb": self.execute_notebook_via_api( input_path=str(self.universe_file), output_path=str(output_path), parameters={ "settings": settings_str, }, ) elif universe_filetype == ".py": self.execute_python_script( input_path=str(self.universe_file), output_path=str(output_path), parameters=settings, ) else: raise ValueError("Universe file must be a .ipynb or .py file.") for warning in w: logger.warning( f"Warning in universe {universe_id}: {warning.message}" ) except Exception as e: error_filename = "E_" + output_filename logger.error(f"Error in universe {universe_id} ({error_filename})") # Rename notebook file to indicate error error_output_path = output_dir / error_filename output_path.rename(error_output_path) if self.stop_on_error: raise e else: logger.exception(e) self.save_error(universe_id, universe_dimensions, e) ``` -------------------------------- ### python -m multiversum aggregate-data options Source: https://simson.io/multiversum/cli Shows the options for the aggregate-data subcommand. ```bash --config PATH Relative path to a TOML, JSON or Python file with a config for the multiverse. Defaults to searching for multiverse.toml, multiverse.json, multiverse.py (in that order). --output-dir PATH Relative path to output directory for the results. --run INTEGER Run number to aggregate data for. Defaults to the latest run. --help Show this message and exit. ``` -------------------------------- ### examine_multiverse() method implementation Source: https://simson.io/multiversum/api/multiverse_analysis Runs the analysis for all universes in the multiverse. Accepts an optional multiverse_grid and n_jobs parameter (default -2). If multiverse_grid is None, uses self.grid or generates a new grid. Handles single-threaded mode (n_jobs=1) and parallel mode using joblib's Parallel and rich_joblib for progress display. ```python def examine_multiverse( self, multiverse_grid: List[Dict[str, Any]] = None, n_jobs: int = -2 ) -> None: """ Run the analysis for all universes in the multiverse. Args: multiverse_grid: A list of dictionaries containing the settings for different universes. n_jobs: The number of jobs to run in parallel. Defaults to -2 (all CPUs but one). Use -1 for all CPUs, positive integers for specific number of CPUs, or 1 to disable parallel processing. Returns: None """ if multiverse_grid is None: multiverse_grid = self.grid or self.generate_grid(save_format="none") n_jobs = n_jobs if n_jobs > 0 else calculate_cpu_count(n_jobs) # Run analysis for all universes with Progress( TextColumn("[progress.description]{task.description}"), SpinnerColumn(), TaskProgressColumn(), BarColumn(bar_width=None), MofNCompleteColumn(), TextColumn("•"), TimeElapsedColumn(), TextColumn("•"), TimeRemainingColumn(), expand=True, ) as progress: task_id = progress.add_task("Running", total=len(multiverse_grid)) if n_jobs == 1: logger.info("Running in single-threaded mode.") for universe_params in multiverse_grid: self.visit_universe(universe_params) progress.update(task_id, advance=1) # Somehow automatic updating is not working in single threaded mode, so we manually refresh progress.refresh() else: logger.info(f"Running in parallel mode (n_jobs = {n_jobs}).") with rich_joblib(progress, task_id): # For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. # Thus for n_jobs = -2, all CPUs but one are used Parallel(n_jobs=n_jobs)( delayed(self.visit_universe)(universe_params) for universe_params in multiverse_grid ) ``` -------------------------------- ### examine_multiverse(multiverse_grid=None, n_jobs=-2) Source: https://simson.io/multiversum/api/multiverse_analysis Runs the analysis for all universes in the multiverse. Accepts an optional grid of universe settings and a parallelization parameter. ```APIDOC ## examine_multiverse(multiverse_grid=None, n_jobs=-2) ### Description Run the analysis for all universes in the multiverse. ### Method examine_multiverse(multiverse_grid=None, n_jobs=-2) ### Parameters - **multiverse_grid** (List[Dict[str, Any]]) - Optional - A list of dictionaries containing the settings for different universes. Defaults to None. - **n_jobs** (int) - Optional - The number of jobs to run in parallel. Defaults to -2 (all CPUs but one). Use -1 for all CPUs, positive integers for specific number of CPUs, or 1 to disable parallel processing. ### Returns - **None** - None ### Example ```python multiverse = Multiverse() multiverse.examine_multiverse(multiverse_grid=[{"param": 1}], n_jobs=2) ``` ``` -------------------------------- ### Generate sub-universes from universe settings Source: https://simson.io/multiversum/api/universe Generates sub-universes by wrapping non-list dimension values with `list_wrap` and passing them to `generate_multiverse_grid`. Returns a list of dictionaries representing the sub-universes. ```python def generate_sub_universes(self) -> List[dict]: """ Generate the sub-universes for the given universe settings. Returns: A list of dictionaries containing the sub-universes. """ # Wrap all non-lists in the universe to make them work with generate_multiverse_grid universe_all_lists = {k: list_wrap(v) for k, v in self.dimensions.items()} # Within-Universe variation return generate_multiverse_grid(universe_all_lists) ``` -------------------------------- ### get_export_file_path(filename, mkdir=False) Source: https://simson.io/multiversum/api/universe Returns the full path for exporting a file from this Universe. Optionally creates the directory if it does not exist. ```APIDOC ## get_export_file_path(filename, mkdir=False) ### Description Get the file path for exporting a file from this Universe. ### Method Function call ### Endpoint Universe.get_export_file_path(filename, mkdir=False) ### Parameters #### Arguments - **filename** (str) - Required - The name of the file to export. - **mkdir** (bool) - Optional - Whether to create the directory if it does not exist. Default: False. ### Returns - **Path** - The full path where the file should be exported. ### Example ```python path = universe.get_export_file_path('output.csv', mkdir=True) ``` ``` -------------------------------- ### get_export_file_path() method implementation Source: https://simson.io/multiversum/api/universe Returns a Path object for the export file. If mkdir is True, it creates the export directory (including parents) if it does not exist. ```python def get_export_file_path(self, filename: str, mkdir: bool = False) -> Path: """ Get the file path for exporting a file from this Universe. Args: filename: The name of the file to export. mkdir: Whether to create the directory if it does not exist. (optional) Returns: Path: The full path where the file should be exported. """ export_dir = ( self.output_dir / "runs" / str(self.run_no) / "exports" / self.universe_id ) if mkdir: export_dir.mkdir(parents=True, exist_ok=True) return export_dir / filename ``` -------------------------------- ### python -m multiversum run usage Source: https://simson.io/multiversum/cli Shows the usage of the run subcommand. ```bash python -m multiversum run [OPTIONS] ```