### Install OCTIS from Source Source: https://octis.readthedocs.io/en/latest/installation Installs OCTIS after obtaining the source code (either by cloning or downloading a tarball). This command executes the setup script to build and install the package. ```bash $ python setup.py install ``` -------------------------------- ### Install OCTIS Stable Release with Pip Source: https://octis.readthedocs.io/en/latest/installation Installs the latest stable version of OCTIS from the Python Package Index (PyPI). This is the recommended installation method. Ensure pip is installed; refer to the Python installation guide if needed. ```bash $ pip install octis ``` -------------------------------- ### Install OCTIS using pip Source: https://octis.readthedocs.io/en/latest/readme Installs the OCTIS library and its dependencies using pip. Ensure you have Python and pip installed. ```bash pip install octis ``` -------------------------------- ### Download OCTIS Source Tarball Source: https://octis.readthedocs.io/en/latest/installation Downloads the OCTIS source code as a tarball from the master branch of the GitHub repository. This is an alternative to cloning the repository. ```bash $ curl -OJL https://github.com/mind-lab/octis/tarball/master ``` -------------------------------- ### Set up Virtual Environment and Install OCTIS Source: https://octis.readthedocs.io/en/latest/contributing Sets up a virtual environment named 'OCTIS' and installs the local copy of OCTIS in development mode using 'setup.py develop'. This command requires virtualenvwrapper to be installed. ```bash $ mkvirtualenv OCTIS $ cd OCTIS/ $ python setup.py develop ``` -------------------------------- ### Clone OCTIS GitHub Repository Source: https://octis.readthedocs.io/en/latest/installation Clones the OCTIS public repository from GitHub to obtain the source code. This method allows direct access to the project's codebase for development or modification. ```bash $ git clone git://github.com/mind-lab/octis ``` -------------------------------- ### ETM Model Setup and Data Preprocessing (Python) Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/ETM This function initializes the ETM model, handling dataset partitioning, vocabulary creation, and data preprocessing. It sets up training, validation, and testing data based on whether partitions are used. It also defines the model's architecture, including hyperparameters like number of topics, hidden sizes, and embedding dimensions, and moves the model to the appropriate device (CPU or GPU). ```python def set_model(self, dataset, hyperparameters): if self.use_partitions: train_data, validation_data, testing_data = ( dataset.get_partitioned_corpus(use_validation=True)) data_corpus_train = [' '.join(i) for i in train_data] data_corpus_test = [' '.join(i) for i in testing_data] data_corpus_val = [' '.join(i) for i in validation_data] vocab = dataset.get_vocabulary() self.vocab = {i: w for i, w in enumerate(vocab)} vocab2id = {w: i for i, w in enumerate(vocab)} (self.train_tokens, self.train_counts, self.test_tokens, self.test_counts, self.valid_tokens, self.valid_counts ) = self.preprocess( vocab2id, data_corpus_train, data_corpus_test, data_corpus_val) else: data_corpus = [' '.join(i) for i in dataset.get_corpus()] vocab = dataset.get_vocabulary() self.vocab = {i: w for i, w in enumerate(vocab)} vocab2id = {w: i for i, w in enumerate(vocab)} self.train_tokens, self.train_counts = self.preprocess( vocab2id, data_corpus, None) self.device = torch.device( "cuda" if torch.cuda.is_available() else "cpu") self.set_default_hyperparameters(hyperparameters) self.load_embeddings() # define model and optimizer self.model = etm.ETM( num_topics=self.hyperparameters['num_topics'], vocab_size=len(self.vocab.keys()), t_hidden_size=int(self.hyperparameters['t_hidden_size']), rho_size=int(self.hyperparameters['rho_size']), emb_size=int(self.hyperparameters['embedding_size']), theta_act=self.hyperparameters['activation'], embeddings=self.embeddings, train_embeddings=self.hyperparameters['train_embeddings'], enc_drop=self.hyperparameters['dropout']).to(self.device) print('model: {}'.format(self.model)) self.optimizer = self.set_optimizer() ``` -------------------------------- ### Launch Hyper-parameter Optimization - Python Source: https://octis.readthedocs.io/en/latest/optimization Launches the hyper-parameter optimization process using the defined model, dataset, metric, and search space. Key parameters control the optimization process, including the number of calls, random starts, and model runs. ```python optimization_result=optimizer.optimize(model, dataset, npmi, search_space, number_of_call=10, n_random_starts=3, model_runs=3, save_name="result", surrogate_model="RF", acq_func="LCB") ``` -------------------------------- ### Get Partitioned Corpus Source: https://octis.readthedocs.io/en/latest/_modules/octis/dataset/dataset Retrieves the corpus partitioned into training, validation, and testing sets. The partitioning logic relies on metadata keys like 'last-training-doc' and 'last-validation-doc'. ```python def get_partitioned_corpus(self, use_validation=True): if "last-training-doc" in self.__metadata: last_training_doc = self.__metadata["last-training-doc"] if use_validation: last_validation_doc = self.__metadata["last-validation-doc"] if self.__corpus is not None and last_training_doc != 0: train_corpus = [] test_corpus = [] validation_corpus = [] for i in range(last_training_doc): train_corpus.append(self.__corpus[i]) for i in range(last_training_doc, last_validation_doc): validation_corpus.append(self.__corpus[i]) for i in range(last_validation_doc, len(self.__corpus)): test_corpus.append(self.__corpus[i]) return train_corpus, validation_corpus, test_corpus else: if self.__corpus is not None and last_training_doc != 0: if "last-validation-doc" in self.__metadata.keys(): last_validation_doc = self.__metadata["last-validation-doc"] else: last_validation_doc = 0 train_corpus = [] test_corpus = [] for i in range(last_training_doc): train_corpus.append(self.__corpus[i]) if last_validation_doc != 0: for i in range(last_validation_doc, len(self.__corpus)): test_corpus.append(self.__corpus[i]) else: for i in range(last_training_doc, len(self.__corpus)): test_corpus.append(self.__corpus[i]) return train_corpus, test_corpus else: return [self.__corpus] ``` -------------------------------- ### ETM Inference or Get Info (Python) Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/ETM This function determines the result based on the 'use_partitions' flag. If partitions are in use, it calls the 'inference' method. Otherwise, it calls the 'get_info' method. The result from the chosen method is then returned. ```python if self.use_partitions: result = self.inference() else: result = self.get_info() return result ``` -------------------------------- ### Run Code Quality and Tests Source: https://octis.readthedocs.io/en/latest/contributing Checks if your changes pass flake8 linting and run the project's tests. It also includes instructions for testing with different Python versions using tox. Flake8 and tox can be installed via pip. ```bash $ flake8 octis tests $ python setup.py test or pytest $ tox ``` -------------------------------- ### Run OCTIS Dashboard Server Source: https://octis.readthedocs.io/en/latest/readme This command initiates the OCTIS dashboard, providing a user-friendly graphical interface for managing experiments. After cloning the repository, running this command from the project directory will launch the server, allowing users to create, monitor, and visualize experiments through their web browser. ```bash python OCTIS/dashboard/server.py ``` -------------------------------- ### Load and Train LDA Model Source: https://octis.readthedocs.io/en/latest/readme Demonstrates loading a custom dataset from a folder and training an LDA model. It highlights the initialization of the Dataset and LDA model objects, followed by the training process. The dataset can be partitioned for separate training and testing or trained on the entire dataset. ```python from octis.models.contextual_lda import ContextualLDAModel dataset = Dataset() dataset.load_custom_dataset_from_folder("dataset_folder") model = ContextualLDAModel(num_topics=25) # Create model model_output = model.train_model(dataset) # Train the model ``` -------------------------------- ### Get Topic Words and Document-Topic Matrix (Python) Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/LDA Helper methods to retrieve the most significant words for each topic and the topic representation of the corpus. These functions are internal to the Octis library. ```python def _get_topics_words(self, topk): """ Return the most significative words for each topic. """ topic_terms = [] for i in range(self.hyperparameters["num_topics"]): topic_words_list = [] for word_tuple in self.trained_model.get_topic_terms(i, topk): topic_words_list.append(self.id2word[word_tuple[0]]) topic_terms.append(topic_words_list) return topic_terms def _get_topic_document_matrix(self): """ Return the topic representation of the corpus """ doc_topic_tuples = [] ``` -------------------------------- ### Define Topic Model and Hyperparameters - Python Source: https://octis.readthedocs.io/en/latest/optimization Defines a Topic Model, specifically LDA (Latent Dirichlet Allocation) in this example, and sets its hyperparameters, such as the number of topics. This model will be used for optimization. ```python from octis.models.LDA import LDA model = LDA() model.hyperparameters.update({"num_topics": 25}) ``` -------------------------------- ### Initialize Optimizer and Load Dataset - Python Source: https://octis.readthedocs.io/en/latest/optimization Initializes the Optimizer class and loads a specified dataset for topic modeling analysis. This is the first step in setting up hyper-parameter optimization. ```python from octis.optimization.optimizer import Optimizer optimizer = Optimizer() from octis.dataset.dataset import Dataset dataset = Dataset() dataset.load("octis/preprocessed_datasets/M10") ``` -------------------------------- ### Train a topic model with OCTIS Source: https://octis.readthedocs.io/en/latest/readme Demonstrates the initial steps to train a topic model. It involves loading a preprocessed dataset and specifying the model, such as LDA. Requires 'Dataset' from 'octis.dataset.dataset' and the specific model class (e.g., 'LDA' from 'octis.models.LDA'). ```python from octis.dataset.dataset import Dataset from octis.models.LDA import LDA # Further steps to set hyperparameters and call train_model() would follow. ``` -------------------------------- ### Perform Inference with the Trained Model Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/CTM Performs inference on test data using the trained CTM model. It asserts that partitioning is enabled and then uses the model's predict method to get results. ```python def inference(self, x_test): assert isinstance(self.use_partitions, bool) and self.use_partitions results = self.model.predict(x_test) return results ``` -------------------------------- ### Initialize Bayesian Optimization Parameters Source: https://octis.readthedocs.io/en/latest/_modules/octis/optimization/optimizer Initializes the parameters for Bayesian Optimization, including model, dataset, search space, and various settings for optimization control, saving, and plotting. It also sets up directories for saving results and models, and performs initial checks on parameters. ```python # Set the attributes if extra_metrics is None: extra_metrics = [] if y0 is None: y0 = [] if x0 is None: x0 = dict() self.model = model self.dataset = dataset self.metric = metric self.search_space = search_space self.extra_metrics = extra_metrics self.optimization_type = optimization_type self.number_of_call = number_of_call self.n_random_starts = n_random_starts self.initial_point_generator = initial_point_generator self.model_runs = model_runs self.surrogate_model = surrogate_model self.kernel = kernel self.acq_func = acq_func self.random_state = random_state self.x0 = x0 self.y0 = y0 self.save_path = save_path self.save_step = save_step self.save_name = save_name self.save_models = save_models self.early_stop = early_stop self.early_step = early_step self.plot_model = plot_model self.plot_best_seen = plot_best_seen self.plot_name = plot_name self.log_scale_plot = log_scale_plot self.topk = topk self.hyperparameters = list(sorted(self.search_space.keys())) self.dict_model_runs = dict() self.number_of_previous_calls = 0 self.current_call = 0 self.time_eval = [] self.name_optimized_metric = metric.__class__.__name__ self.dict_model_runs[self.name_optimized_metric] = dict() # Info about extra metrics i = 0 self.extra_metric_names = [] for extra_metric in extra_metrics: self.extra_metric_names.append( str(i) + '_' + extra_metric.__class__.__name__) self.dict_model_runs[self.extra_metric_names[i]] = dict() i = i + 1 # Control about the correctness of BO parameters if self._check_bo_parameters() == -1: print("ERROR: wrong initialitation of BO parameters") return None # Create the directory where the results are saved Path(self.save_path).mkdir(parents=True, exist_ok=True) # Initialize the directories about model_runs if self.save_models: self.model_path_models = self.save_path + "models/" Path(self.model_path_models).mkdir(parents=True, exist_ok=True) # Choice of the optimizer opt = choose_optimizer(self) # Perform Bayesian Optimization results = self._optimization_loop(opt) return results ``` -------------------------------- ### Get NMF_scikit Hyperparameters Information Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/NMF_scikit Retrieves the default hyperparameter information for the NMF_scikit model. This function utilizes predefined defaults from the octis.configuration module, providing details on available parameters and their characteristics. ```python [docs] def hyperparameters_info(self): """ Returns hyperparameters informations """ return defaults.NMF_scikit_hyperparameters_info ``` -------------------------------- ### Clone OCTIS Repository Source: https://octis.readthedocs.io/en/latest/contributing Clones the OCTIS repository to your local machine. This is the first step in setting up your fork for local development. ```bash $ git clone git@github.com:your_name_here/OCTIS.git ``` -------------------------------- ### Preprocess a dataset using OCTIS Source: https://octis.readthedocs.io/en/latest/readme Initializes and uses the Preprocessing class to preprocess a dataset from specified corpus and labels files. The preprocessed dataset can then be saved. Requires 'os', 'string', and 'Preprocessing' from 'octis.preprocessing.preprocessing'. ```python import os import string from octis.preprocessing.preprocessing import Preprocessing os.chdir(os.path.pardir) preprocessor = Preprocessing(vocabulary=None, max_features=None, remove_punctuation=True, punctuation=string.punctuation, lemmatize=True, stopword_list='english', min_chars=1, min_words_docs=0) dataset = preprocessor.preprocess_dataset(documents_path=r'..\corpus.txt', labels_path=r'..\labels.txt') dataset.save('hello_dataset') ``` -------------------------------- ### Determine Early Stopping Condition for Optimization Source: https://octis.readthedocs.io/en/latest/_modules/octis/optimization/optimizer_tool Determines if the optimization process should stop early based on a lack of improvement over a specified number of iterations. It uses the convergence trace and requires a minimum number of total iterations including random starting points. ```python def early_condition(values, n_stop, n_random): """ Compute the early-stop criterium to stop or not the optimization. :param values: values obtained by Bayesian Optimization :type values: list :param n_stop: Range of points without improvement :type n_stop: int :param n_random: Random starting points :type n_random: int :return: 'True' if early stop condition reached, 'False' otherwise :rtype: bool """ n_min_len = n_stop + n_random if len(values) >= n_min_len: values = convergence_res(values, optimization_type="minimize") worst = values[len(values) - n_stop] best = values[-1] diff = worst - best if diff == 0: return True return False ``` -------------------------------- ### Load a custom dataset from a folder Source: https://octis.readthedocs.io/en/latest/readme Loads a custom preprocessed dataset from a specified folder. The dataset must be in a specific format with corpus and vocabulary files. Requires the 'Dataset' class. ```python from octis.dataset.dataset import Dataset dataset = Dataset() dataset.load_custom_dataset_from_folder("../path/to/the/dataset/folder") ``` -------------------------------- ### Compute SVM Output and Recall Score in Python Source: https://octis.readthedocs.io/en/latest/_modules/octis/evaluation_metrics/classification_metrics Computes the recall score for a classification model. This involves processing model outputs to get SVM results and then applying scikit-learn's recall_score function. The 'average' parameter influences the averaging method for recall. ```python test_labels, predicted_test_labels, self.same_svm = compute_SVM_output( model_output, self, super()) return recall_score( test_labels, predicted_test_labels, average=self.average) ``` -------------------------------- ### Initialize Dataset Object Source: https://octis.readthedocs.io/en/latest/_modules/octis/dataset/dataset Initializes a Dataset object. Parameters like corpus, vocabulary, labels, and metadata are optional, allowing for flexible dataset creation or loading. ```python def __init__(self, corpus=None, vocabulary=None, labels=None, metadata=None, document_indexes=None): """ Initialize a dataset, parameters are optional if you want to load a dataset, initialize this class with default values and use the load method Parameters ---------- corpus : corpus of the dataset vocabulary : vocabulary of the dataset labels : labels of the dataset metadata : metadata of the dataset """ self.__corpus = corpus self.__vocabulary = vocabulary self.__metadata = metadata self.__labels = labels self.__original_indexes = document_indexes self.dataset_path = None self.is_cached = False ``` -------------------------------- ### Import Utility Libraries for Optimizer Source: https://octis.readthedocs.io/en/latest/_modules/octis/optimization/optimizer Imports essential utility modules for the optimizer, including JSON handling, time, path operations, numerical computation, and specialized libraries for optimization and Gaussian processes. ```python # Utils import json import time from pathlib import Path import numpy as np # utils from skopt and sklearn from sklearn.gaussian_process.kernels import * from skopt.space.space import * from octis.dataset.dataset import Dataset ``` -------------------------------- ### Compute Early Stopping Criterion Source: https://octis.readthedocs.io/en/latest/modules Determines if the optimization process should stop based on the lack of improvement over a certain number of iterations. It takes the history of obtained values, the required number of iterations without improvement, and the number of random starting points as input. Returns a boolean indicating whether to stop. ```python octis.optimization.optimizer_tool.early_condition(_values_ , _n_stop_ , _n_random_) ``` -------------------------------- ### Hyperparameter Optimization Search Space and Initialization Source: https://octis.readthedocs.io/en/latest/readme This Python code demonstrates how to define the search space for hyperparameter optimization using scikit-optimize's Real type. It then initializes the OCTIS Optimizer with a model, dataset, evaluation metric, and the defined search space. The optimization process is configured with the number of iterations and model runs. ```python from octis.optimization.optimizer import Optimizer from skopt.space.space import Real # Define the search space. To see which hyperparameters to optimize, see the topic model's initialization signature search_space = {"alpha": Real(low=0.001, high=5.0), "eta": Real(low=0.001, high=5.0)} # Initialize an optimizer object and start the optimization. optimizer=Optimizer() optResult=optimizer.optimize(model, dataset, eval_metric, search_space, save_path="../results" # path to store the results number_of_call=30, # number of optimization iterations model_runs=5) # number of runs of the topic model #save the results of th optimization in a csv file optResult.save_to_csv("results.csv") ``` -------------------------------- ### Train NMF_scikit Model and Get Output Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/NMF_scikit Trains the NMF_scikit model using a given dataset and optional hyperparameters. It preprocesses the corpus using TF-IDF vectorization and returns a dictionary containing topic information, the topic-word matrix, and the topic-document matrix. Dependencies include TfidfVectorizer and scikit-learn's NMF. ```python [docs] def train_model(self, dataset, hyperparameters=None, top_words=10): """ Train the model and return output Parameters ---------- dataset : dataset to use to build the model hyperparameters : hyperparameters to build the model top_words : if greather than 0 returns the most significant words for each topic in the output Default True Returns ------- result : dictionary with up to 3 entries, 'topics', 'topic-word-matrix' and 'topic-document-matrix' """ if hyperparameters is None: hyperparameters = {} if self.id2word is None or self.id_corpus is None: vectorizer = TfidfVectorizer( min_df=0.0, token_pattern=r"(?u)\b[\w|\-]+\b", vocabulary=dataset.get_vocabulary()) if self.use_partitions: partition = dataset.get_partitioned_corpus( use_validation=False) corpus = partition[0] else: corpus = dataset.get_corpus() real_corpus = [" ".join(document) for document in corpus] X = vectorizer.fit_transform(real_corpus) self.id2word = {i: k for i, k in enumerate( vectorizer.get_feature_names())} if self.use_partitions: test_corpus = [] for document in partition[1]: test_corpus.append(" ".join(document)) Y = vectorizer.transform(test_corpus) self.id_corpus = X self.new_corpus = Y else: self.id_corpus = X #hyperparameters["corpus"] = self.id_corpus #hyperparameters["id2word"] = self.id2word self.hyperparameters.update(hyperparameters) model = NMF( ``` -------------------------------- ### Get Top Words for Topics (Python) Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/NMF_scikit This function identifies and returns the top specified number of words for each topic based on their weights in the topic-word matrix (H). It requires a mapping from word indices to actual words (id2word). The input H is expected to be a numpy array where rows represent topics and columns represent word weights. ```python def get_topics(self, H, top_words): topic_list = [] for topic in H: words_list = sorted( list(enumerate(topic)), key=lambda x: x[1]) topk = [tup[0] for tup in words_list[0:top_words]] topic_list.append([self.id2word[i] for i in topk]) return topic_list ``` -------------------------------- ### AbstractModel Training and Output in Python Source: https://octis.readthedocs.io/en/latest/modules Illustrates the training process for a topic model using the AbstractModel class. It takes a dataset and hyperparameters, returning a dictionary with model outputs like topics and matrices. ```python class AbstractModel: def train_model(self, dataset, hyperparameters, top_words=10): """Train the model.""" pass ``` -------------------------------- ### Fetch Dataset from Cache or Download (Python) Source: https://octis.readthedocs.io/en/latest/_modules/octis/dataset/dataset Loads a dataset by name, either from a local cache or by downloading it if missing. It handles decompression and deserialization of cached data using pickle and zlib. Dependencies include os, codecs, pickle, and custom functions like get_data_home, _pkl_filepath, and download_dataset. ```python def fetch_dataset(self, dataset_name, data_home=None, download_if_missing=True): """Load the filenames and data from a dataset. Parameters ---------- dataset_name: name of the dataset to download or retrieve data_home : optional, default: None Specify a download and cache folder for the datasets. If None, all data is stored in '~/octis' subfolders. download_if_missing : optional, True by default If False, raise an IOError if the data is not locally available instead of trying to download the data from the source site. """ data_home = get_data_home(data_home=data_home) cache_path = _pkl_filepath(data_home, dataset_name + ".pkz") dataset_home = join(data_home, dataset_name) cache = None if exists(cache_path): try: with open(cache_path, 'rb') as f: compressed_content = f.read() uncompressed_content = codecs.decode( compressed_content, 'zlib_codec') cache = pickle.loads(uncompressed_content) except Exception as e: print(80 * '_') print('Cache loading failed') print(80 * '_') print(e) if cache is None: if download_if_missing: cache = download_dataset( dataset_name, target_dir=dataset_home, cache_path=cache_path) else: raise IOError(dataset_name + ' dataset not found') self.is_cached = True self.__corpus = [d.split() for d in cache["corpus"]] self.__vocabulary = cache["vocabulary"] self.__metadata = cache["metadata"] self.dataset_path = cache_path self.__labels = cache["labels"] ``` -------------------------------- ### Load Dataset using octis.dataset.dataset.Dataset Source: https://octis.readthedocs.io/en/latest/modules Loads dataset filenames and data. It can download datasets if they are missing and allows specifying a custom data home directory. Handles both existing local data and remote downloads. ```python from octis.dataset.dataset import Dataset dataset_manager = Dataset() # Example: Load a dataset named '20newsgroup' data = dataset_manager.fetch_dataset('20newsgroup') # Example: Load with a custom data home and disable download if missing try: data = dataset_manager.fetch_dataset('20newsgroup', data_home='/path/to/datasets', download_if_missing=False) except IOError as e: print(f"Dataset not found locally: {e}") ``` -------------------------------- ### Load a preprocessed dataset from OCTIS Source: https://octis.readthedocs.io/en/latest/readme Fetches a preprocessed dataset from OCTIS. Requires the 'Dataset' class from 'octis.dataset.dataset'. The dataset name is case-sensitive. ```python from octis.dataset.dataset import Dataset dataset = Dataset() dataset.fetch_dataset("20NewsGroup") ``` -------------------------------- ### Load Topic Model for Optimization with Python Source: https://octis.readthedocs.io/en/latest/_modules/octis/optimization/optimizer_tool Loads a topic model dynamically for resuming optimization processes. It takes a dictionary containing model attributes, name, and partitioning settings. The function constructs the module path, imports the specified model class, instantiates it, and updates its hyperparameters and partitioning settings. ```python import os def load_model(optimization_object): """ Load the topic model for the resume of the optimization :param optimization_object: dictionary of optimization attributes saved in the json file :type optimization_object: dict :return: topic model used during the BO. :rtype: object model """ model_parameters = optimization_object['model_attributes'] use_partitioning = optimization_object['use_partitioning'] model_name = optimization_object['model_name'] module_path = os.path.join(framework_path, "models") module_path = os.path.join(module_path, model_name + ".py") model = importClass(model_name, model_name, module_path) model_instance = model() model_instance.hyperparameters.update(model_parameters) model_instance.use_partitions = use_partitioning return model_instance ``` -------------------------------- ### Run OCTIS Local Dashboard Server Source: https://octis.readthedocs.io/en/latest/dashboard Command to launch the OCTIS local dashboard server. It requires specifying the Python script to run and optionally allows configuration of the port and dashboard state file path. The default port is 5000, and the default dashboard state path is the current directory. ```bash python OCTIS/dashboard/server.py --port [port number] --dashboardState [path to dashboard state file] ``` -------------------------------- ### LogOddsRatio Metric Initialization and Scoring Source: https://octis.readthedocs.io/en/latest/_modules/octis/evaluation_metrics/diversity_metrics Initializes the LogOddsRatio metric, which calculates the Log Odds Ratio between topic-word distributions. The score method iterates through pairs of topic-word matrices and computes the average Log Odds Ratio. ```python class LogOddsRatio(AbstractMetric): def __init__(self): """ Initialize metric Log Odds Ratio """ super().__init__() def score(self, model_output): beta = model_output['topic-word-matrix'] lor = 0 count = 0 for i, j in itertools.combinations(range(len(beta)), 2): lor += _LOR(beta[i], beta[j]) count += 1 return lor / count ``` -------------------------------- ### LDA Partitioning Configuration Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/LDA Configures how data partitioning is handled for the LDA model, specifying whether to use partitions and if updates should include test data. ```python def partitioning(self, use_partitions, update_with_test=False): """ ``` -------------------------------- ### Dataset Handling API Source: https://octis.readthedocs.io/en/latest/modules APIs for managing datasets, including loading, saving, and fetching pre-defined datasets. ```APIDOC ## Dataset Class ### Description Handles a dataset and offers methods to access, save and edit the dataset data. ### Methods - **`Dataset(_corpus=None, _vocabulary=None, _labels=None, _metadata=None, _document_indexes=None)`**: Constructor for the Dataset class. ### fetch_dataset #### Description Load the filenames and data from a dataset. #### Parameters - **`dataset_name`** (str) - Name of the dataset to download or retrieve. - **`data_home`** (str, optional) - Specify a download and cache folder for the datasets. Defaults to `None` (all data stored in `~/octis`). - **`download_if_missing`** (bool, optional) - If `False`, raise an `IOError` if the data is not locally available instead of trying to download it. Defaults to `True`. ### load_custom_dataset_from_folder #### Description Loads all the dataset from a folder. #### Parameters - **`path`** (str) - Path of the folder to read. - **`multilabel`** (bool, optional) - Whether the dataset is multilabel. Defaults to `False`. ### save #### Description Saves all the dataset info in a folder. #### Parameters - **`path`** (str) - Path to the folder in which files are saved. If the folder doesn't exist it will be created. - **`multilabel`** (bool, optional) - Whether the dataset is multilabel. Defaults to `False`. ``` -------------------------------- ### Load Custom Dataset from Folder (Python) Source: https://octis.readthedocs.io/en/latest/_modules/octis/dataset/dataset Loads a dataset from a specified folder, parsing corpus, labels, and metadata. It handles different dataset structures, including multilabel formats, and attempts to load vocabulary and document indexes if they exist. Dependencies include pandas and os.path.exists. ```python def load_custom_dataset_from_folder(self, path, multilabel=False): """ Loads all the dataset from a folder Parameters ---------- path : path of the folder to read """ self.dataset_path = path try: if exists(self.dataset_path + "/metadata.json"): self._load_metadata(self.dataset_path + "/metadata.json") else: self.__metadata = dict() df = pd.read_csv( self.dataset_path + "/corpus.tsv", sep='\t', header=None) if len(df.keys()) > 1: # just make sure docs are sorted in the right way (train - val - test) final_df = pd.concat( [df[df[1] == 'train'], df[df[1] == 'val'], df[df[1] == 'test']]) self.__metadata['last-training-doc'] = len( final_df[final_df[1] == 'train']) self.__metadata['last-validation-doc'] = len( final_df[final_df[1] == 'val']) + len( final_df[final_df[1] == 'train']) self.__corpus = [d.split() for d in final_df[0].tolist()] if len(final_df.keys()) > 2: if multilabel: self.__labels = [ doc.split() for doc in final_df[2].tolist()] else: self.__labels = final_df[2].tolist() else: self.__corpus = [d.split() for d in df[0].tolist()] self.__metadata['last-training-doc'] = len(df[0]) if exists(self.dataset_path + "/vocabulary.txt"): self._load_vocabulary(self.dataset_path + "/vocabulary.txt") else: vocab = set() for d in self.__corpus: for w in set(d): vocab.add(w) self.__vocabulary = list(vocab) if exists(self.dataset_path + "/indexes.txt"): self._load_document_indexes(self.dataset_path + "/indexes.txt") except: raise Exception("error in loading the dataset:" + self.dataset_path) ``` -------------------------------- ### Initialize LDA Model Source: https://octis.readthedocs.io/en/latest/_modules/octis/models/LDA Initializes the Latent Dirichlet Allocation (LDA) model with various configuration parameters. Supports distributed computing and custom priors for topic-word distributions. ```python from octis.models.model import AbstractModel import numpy as np from gensim.models import ldamodel import gensim.corpora as corpora import octis.configuration.citations as citations import octis.configuration.defaults as defaults class LDA(AbstractModel): id2word = None id_corpus = None use_partitions = True update_with_test = False def __init__( self, num_topics=100, distributed=False, chunksize=2000, passes=1, update_every=1, alpha="symmetric", eta=None, decay=0.5, offset=1.0, eval_every=10, iterations=50, gamma_threshold=0.001, random_state=None ): """ Initialize LDA model Parameters ---------- num_topics (int, optional) – The number of requested latent topics to be extracted from the training corpus. distributed (bool, optional) – Whether distributed computing should be used to accelerate training. chunksize (int, optional) – Number of documents to be used in each training chunk. passes (int, optional) – Number of passes through the corpus during training. update_every (int, optional) – Number of documents to be iterated through for each update. Set to 0 for batch learning, > 1 for online iterative learning. alpha ({numpy.ndarray, str}, optional) – Can be set to an 1D array of length equal to the number of expected topics that expresses our a-priori belief for the each topics’ probability. Alternatively default prior selecting strategies can be employed by supplying a string: ’asymmetric’: Uses a fixed normalized asymmetric prior of 1.0 / topicno. ’auto’: Learns an asymmetric prior from the corpus (not available if distributed==True). eta ({float, np.array, str}, optional) – A-priori belief on word probability, this can be: scalar for a symmetric prior over topic/word probability, vector of length num_words to denote an asymmetric user defined probability for each word, matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination, the string ‘auto’ to learn the asymmetric prior from the data. decay (float, optional) – A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten when each new document is examined. offset (float, optional) – Hyper-parameter that controls how much we will slow down the first steps the first few iterations. eval_every (int, optional) – Log perplexity is estimated every that many updates. Setting this to one slows down training by ~2x. iterations (int, optional) – Maximum number of iterations through the corpus when inferring the topic distribution of a corpus. gamma_threshold (float, optional) – Minimum change in the value of the gamma parameters to continue iterating. random_state ({np.random.RandomState, int}, optional) – Either a randomState object or a seed to generate one.s Useful for reproducibility. """ super().__init__() self.hyperparameters = dict() self.hyperparameters["num_topics"] = num_topics self.hyperparameters["distributed"] = distributed self.hyperparameters["chunksize"] = chunksize self.hyperparameters["passes"] = passes self.hyperparameters["update_every"] = update_every self.hyperparameters["alpha"] = alpha self.hyperparameters["eta"] = eta self.hyperparameters["decay"] = decay self.hyperparameters["offset"] = offset self.hyperparameters["eval_every"] = eval_every self.hyperparameters["iterations"] = iterations self.hyperparameters["gamma_threshold"] = gamma_threshold self.hyperparameters["random_state"] = random_state ``` -------------------------------- ### Select Evaluation Metric with Python Source: https://octis.readthedocs.io/en/latest/_modules/octis/optimization/optimizer_tool Selects an evaluation metric dynamically based on its name and parameters. It constructs the module path for the metric, imports the corresponding class, and instantiates it with the provided parameters. This function relies on `defaults.metric_parameters` to find the correct module. ```python import os def select_metric(metric_parameters, metric_name): """ Select the metric for the resume of the optimization :param metric_parameters: metric parameters :type metric_parameters: list :param metric_name: name of the metric :type metric_name: str :return: metric :rtype: metric object """ module_path = os.path.join(framework_path, "evaluation_metrics") module_name = defaults.metric_parameters[metric_name]["module"] module_path = os.path.join(module_path, module_name + ".py") Metric = importClass(metric_name, metric_name, module_path) metric = Metric(**metric_parameters) return metric ``` -------------------------------- ### Optimizer Class - resume_optimization Method Source: https://octis.readthedocs.io/en/latest/modules The `resume_optimization` method allows restarting a Bayesian Optimization process from a previously saved state. It takes the path to a JSON file containing the optimization results and an optional number of extra evaluations to perform. ```APIDOC ## POST /optimizer/resume ### Description Restarts the Bayesian Optimization process from a saved JSON file. ### Method POST ### Endpoint /optimizer/resume ### Parameters #### Request Body - **name_path** (str) - Required - The path to the JSON file containing the previous optimization results. - **extra_evaluations** (int, optional) - The number of additional iterations to perform. ### Request Example ```json { "name_path": "/path/to/results.json", "extra_evaluations": 20 } ``` ### Response #### Success Response (200) - **optimization_results** (object) - An object containing the results of the resumed optimization. ``` -------------------------------- ### Choose Optimizer Surrogate Model with Python Source: https://octis.readthedocs.io/en/latest/_modules/octis/optimization/optimizer_tool Configures and returns a surrogate model for Bayesian Optimization. It supports Random Forest ('RF'), Extra Trees ('ET'), and Gaussian Process ('GP') regressors, as well as Random Search ('RS'). The function sets up the `skopt.Optimizer` with the chosen estimator, acquisition function, and other relevant parameters. ```python from skopt.learning import ( GaussianProcessRegressor, RandomForestRegressor, ExtraTreesRegressor) from skopt import Optimizer as skopt_optimizer from skopt.utils import dimensions_aslist def choose_optimizer(optimizer): """ Choose a surrogate model for Bayesian Optimization :param optimizer: list of setting of the BO experiment :type optimizer: Optimizer :return: surrogate model :rtype: scikit object """ params_space_list = dimensions_aslist(optimizer.search_space) estimator = None # Choice of the surrogate model # Random forest if optimizer.surrogate_model == "RF": estimator = RandomForestRegressor( n_estimators=100, min_samples_leaf=3, random_state=optimizer.random_state) # Extra Tree elif optimizer.surrogate_model == "ET": estimator = ExtraTreesRegressor( n_estimators=100, min_samples_leaf=3, random_state=optimizer.random_state) # GP Minimize elif optimizer.surrogate_model == "GP": estimator = GaussianProcessRegressor( kernel=optimizer.kernel, random_state=optimizer.random_state) # Random Search elif optimizer.surrogate_model == "RS": estimator = "dummy" if estimator == "dummy": opt = skopt_optimizer( params_space_list, base_estimator=estimator, acq_func=optimizer.acq_func, acq_optimizer='sampling', initial_point_generator=optimizer.initial_point_generator, random_state=optimizer.random_state) else: opt = skopt_optimizer( params_space_list, base_estimator=estimator, acq_func=optimizer.acq_func, acq_optimizer='sampling', n_initial_points=optimizer.n_random_starts, initial_point_generator=optimizer.initial_point_generator, # work only for version skopt 8.0!!! acq_optimizer_kwargs={ "n_points": 10000, "n_restarts_optimizer": 5, "n_jobs": 1}, acq_func_kwargs={"xi": 0.01, "kappa": 1.96}, random_state=optimizer.random_state) return opt ```