### Install Melusine from Source

Source: https://melusine.readthedocs.io/en/latest/installation.html

Install Melusine after downloading or cloning the source code using the setup.py script.

```bash
$ python setup.py install
```

--------------------------------

### TransformerScheduler Initialization Example

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/utils/transformer_scheduler.html

Demonstrates how to initialize the TransformerScheduler with a list of functions, their arguments, and expected return columns. This setup is useful for creating custom data transformation pipelines.

```python
from melusine.utils.transformer_scheduler import TransformerScheduler

MelusineTransformer = TransformerScheduler(
    functions_scheduler=[
        (my_function_1, (argument1, argument2), ['return_col_A']),
        (my_function_2, None, ['return_col_B', 'return_col_C'])
        (my_function_3, (), ['return_col_D'])
    ])
```

--------------------------------

### Set Up Virtual Environment and Install Melusine

Source: https://melusine.readthedocs.io/en/latest/contributing.html

Install your local copy of Melusine into a virtual environment using virtualenvwrapper.

```bash
$ mkvirtualenv melusine
$ cd melusine/
$ python setup.py develop

```

--------------------------------

### Install Melusine with Optional Dependencies

Source: https://melusine.readthedocs.io/en/latest/installation.html

Install Melusine with specific optional dependencies for features like visualization ('viz'), Outlook Exchange integration ('exchange'), or training BERT-like models ('transformers'). Use 'all' to install all optional dependencies.

```bash
pip install melusine[viz]
```

```bash
pip install melusine[exchange]
```

```bash
pip install melusine[transformers]
```

```bash
pip install melusine[all]
```

--------------------------------

### Get Configuration Dictionary

Source: https://melusine.readthedocs.io/en/latest/usage.html

Retrieves and prints the configuration as a dictionary.

```python
conf_dict = conf.get_config_file()
print(conf_dict)  # will print the json
```

--------------------------------

### Install Melusine Stable Release

Source: https://melusine.readthedocs.io/en/latest/installation.html

Use this command to install the latest stable version of Melusine. This is the recommended method for most users.

```bash
$ pip install melusine
```

--------------------------------

### Define TransformerScheduler with Functions

Source: https://melusine.readthedocs.io/en/latest/melusine.utils.html

Instantiate TransformerScheduler with a list of functions, their arguments, and expected return columns. This example shows how to define a custom transformer pipeline.

```python
MelusineTransformer = TransformerScheduler(
    functions_scheduler=[
        (my_function_1, (argument1, argument2), ['return_col_A']),
        (my_function_2, None, ['return_col_B', 'return_col_C'])
        (my_function_3, (), ['return_col_D'])
    ])
```

--------------------------------

### Keyword Extraction Example

Source: https://melusine.readthedocs.io/en/latest/usage.html

Demonstrates the use of the KeywordGenerator to extract relevant keywords from text data, prioritizing rarer words.

```python
keywords_generator.transform(df_zoo)
```

--------------------------------

### Load Example Email Data

Source: https://melusine.readthedocs.io/en/latest/usage.html

Load a toy DataFrame containing anonymized emails for testing and examples. This function is part of the melusine utility functions.

```python
from melusine.melusine.utils.data_loader import load_email_data

df_emails = load_email_data()
df_emails.head()
```

--------------------------------

### Example Melusine Pre-processing Pipeline

Source: https://melusine.readthedocs.io/en/latest/usage.html

Demonstrates a complete pre-processing pipeline using multiple TransformerScheduler instances for managing transfer/reply status, building email history, and structuring email content, all chained within a Scikit-learn Pipeline.

```python
from sklearn.pipeline import Pipeline
from melusine.utils.transformer_scheduler import TransformerScheduler
from melusine.prepare_email.manage_transfer_reply import add_boolean_answer, add_boolean_transfer
from melusine.prepare_email.build_historic import build_historic
from melusine.prepare_email.mail_segmenting import structure_email

ManageTransferReply = TransformerScheduler(
    functions_scheduler=[
        (add_boolean_answer, None, ['is_answer']),
        (add_boolean_transfer, None, ['is_transfer'])
])

HistoricBuilder = TransformerScheduler(
    functions_scheduler=[
        (build_historic, None, ['structured_historic']),
])

Segmenting = TransformerScheduler(
    functions_scheduler=[
        (structure_email, None, ['structured_body'])
])

prepare_data_pipeline = Pipeline([
    ('ManageTransferReply', ManageTransferReply),
    ('HistoricBuilder', HistoricBuilder),
    ('Segmenting', Segmenting),
])

df_emails = prepare_data_pipeline.fit_transform(df_emails)

```

--------------------------------

### Load Transformers Tokenizer

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/models/train.html

Loads a tokenizer from the transformers library. Raises an error if the transformers library is not installed or if the specified tokenizer is not supported.

```python
from transformers import CamembertTokenizer

self.tokenizer = CamembertTokenizer.from_pretrained(self.bert_tokenizer)
```

```python
from transformers import XLMTokenizer

self.tokenizer = XLMTokenizer.from_pretrained(self.bert_tokenizer)
```

--------------------------------

### Get Embedding Matrix from Pretrained Embeddings

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/models/train.html

Prepares the embedding matrix using a pre-trained embedding. Sets the vocabulary and initializes the matrix with pre-trained word vectors.

```python
    def _get_embedding_matrix(self):
        """Prepares the embedding matrix to be used as an input for
        the neural network model.
        The vocabulary of the NN is those of the pretrained embedding
        """
        pretrained_embedding = self.pretrained_embedding
        self.vocabulary = pretrained_embedding.embedding.index_to_key
        vocab_size = len(self.vocabulary)
        vector_dim = pretrained_embedding.embedding.vector_size
        embedding_matrix = np.zeros((vocab_size + 2, vector_dim))
        for index, word in enumerate(self.vocabulary):
            if word not in ["PAD", "UNK"]:
                embedding_matrix[index + 2, :] = pretrained_embedding.embedding[word]
        embedding_matrix[1, :] = np.mean(embedding_matrix, axis=0)

        self.vocabulary.insert(0, "PAD")
        self.vocabulary.insert(1, "UNK")

        self.embedding_matrix = embedding_matrix
        pass
```

--------------------------------

### Get Index Transitions

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/build_historic.html

Returns a list of indexes defining the transitions between different messages in an email. It processes the email body using a list of regular expressions to find transition points.

```python
def _get_index_transitions(email_body):
    """Returns list of indexes defining the transitions between
    different messages in an email."""
    index = []
    for regex in regex_transition_list:
        for match in re.finditer(regex, email_body, flags=re.S):
            idx = (match.start(), match.end())
            index.append(idx)

    index = [(0, 0)] + index
    index = index + [(len(email_body), len(email_body))]
    index = sorted(list(set(index)))
    index = __filter_overlap(index)
    nb_parts = len(index) - 1

    return index, nb_parts
```

--------------------------------

### Email Classification with NeuralModel

Source: https://melusine.readthedocs.io/en/latest/usage.html

A minimal working example for email classification using the NeuralModel class. Requires a pandas DataFrame with 'label' and 'clean_text' columns, and a trained word embedding model.

```python
import pandas as pd
```

--------------------------------

### Initialize TransformerScheduler for Email Transfers

Source: https://melusine.readthedocs.io/en/latest/usage.html

Instantiate TransformerScheduler to manage a sequence of functions for processing email transfers and replies. This setup defines the order and conditions for applying specific transformations.

```python
from melusine.utils.transformer_scheduler import TransformerScheduler
from melusine.prepare_email.manage_transfer_reply import \
    check_mail_begin_by_transfer, update_info_for_transfer_mail, add_boolean_answer, add_boolean_transfer

# Transformer object to manage transfers and replies
ManageTransferReply = TransformerScheduler(
    functions_scheduler=[
        (check_mail_begin_by_transfer, None, ['is_begin_by_transfer']),
        (update_info_for_transfer_mail, None, None),
        (add_boolean_answer, None, ['is_answer']),
        (add_boolean_transfer, None, ['is_transfer'])
    ]
)
```

--------------------------------

### Check Mail Begin by Transfer - melusine.prepare_email.manage_transfer_reply

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/manage_transfer_reply.html

Computes a boolean Series indicating if an email body starts with a transfer regex. Use with pandas `apply` for single samples or entire DataFrames.

```python
import re
from melusine import config

regex_transfer_header = config["regex"]["manage_transfer_reply"]["transfer_header"]
regex_answer_header = config["regex"]["manage_transfer_reply"]["answer_header"]
regex_begin_transfer = config["regex"]["manage_transfer_reply"]["begin_transfer"]
regex_begin_transfer_cons = config["regex"]["manage_transfer_reply"]["begin_transfer_cons"]
regex_extract_from = config["regex"]["manage_transfer_reply"]["extract_from"]
regex_extract_to = config["regex"]["manage_transfer_reply"]["extract_to"]
regex_extract_date = config["regex"]["manage_transfer_reply"]["extract_date"]
regex_extract_header = config["regex"]["manage_transfer_reply"]["extract_header"]



[docs]def check_mail_begin_by_transfer(row):
    """Compute boolean Series which return True if the "body" starts with given
    regex 'begin_transfer', False if not.

    To be used with methods such as: `apply(func, axis=1)` or
    `apply_by_multiprocessing(func, axis=1, **kwargs)`.

    Parameters
    ----------
    row : row of pd.Dataframe, columns ['body']

    Returns
    -------
    pd.Series

    Examples
    --------
        >>> import pandas as pd
        >>> data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
        >>> # data contains a 'body' column

        >>> from melusine.prepare_email.manage_transfer_reply import check_mail_begin_by_transfer
        >>> check_mail_begin_by_transfer(data.iloc[0])  # apply for 1 sample
        >>> data.apply(check_mail_begin_by_transfer, axis=1)  # apply to all samples

    """
    is_begin_by_transfer = False
    try:
        if re.search(regex_begin_transfer, row["body"]):
            is_begin_by_transfer = True
        if re.search(regex_begin_transfer_cons, row["body"]):
            is_begin_by_transfer = True
    except Exception:
        pass

    return is_begin_by_transfer

```

--------------------------------

### Tokenizer: Initialize with configuration

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/nlp_tools/tokenizer.html

Initializes the Tokenizer, setting up the input/output columns and configuring the Normalizer, RegexTokenizer, and FlashtextTokenFlagger based on melusine.config.

```python
def __init__(
        self,
        input_column="text",
        output_column="tokens",
        stop_removal=True,
    ):
        self.input_column = input_column
        self.output_column = output_column

        self.normalizer = Normalizer(
            form=config["normalizer"]["form"],
            lowercase=config["normalizer"]["lowercase"],
        )

        if stop_removal:
            stopwords = config["tokenizer"]["stopwords"]
        else:
            stopwords = []
        self.regex_tokenizer = RegexTokenizer(
            stopwords=stopwords,
            tokenizer_regex=config["tokenizer"]["tokenizer_regex"],
        )
        self.token_flagger = FlashtextTokenFlagger(
            token_flags=config["token_flagger"]["token_flags"]
        )
```

--------------------------------

### Load Configuration from Path

Source: https://melusine.readthedocs.io/en/latest/melusine.config.html

Loads configuration settings from YML or JSON files located within a specified directory path.

```python
load_conf_from_path(_config_dir_path : str_) → Dict[str, Any][source]
    
    Given a directory path Parameters ———- config_dir_path: str
> Path to a directory containing YML or JSON conf files 

conf: dict
    
Loaded config dict 

```

--------------------------------

### Initialize Configuration Reader

Source: https://melusine.readthedocs.io/en/latest/usage.html

Demonstrates how to initialize the ConfigJsonReader to manage Melusine's configuration settings.

```python
from melusine.config.config import ConfigJsonReader

conf = ConfigJsonReader()
```

--------------------------------

### Set New Configuration Path

Source: https://melusine.readthedocs.io/en/latest/usage.html

Sets a new path for the configuration file.

```python
conf.set_config_path(file_path='my/path/conf.json')
```

--------------------------------

### Clone Melusine Repository

Source: https://melusine.readthedocs.io/en/latest/installation.html

Clone the Melusine public repository from GitHub to get the source code.

```bash
$ git clone https://github.com/MAIF/melusine
```

--------------------------------

### Get TF-IDF Weights

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/summarizer/keywords_generator.html

Applies maximum weights to specified keywords within the TF-IDF vector, scaling them by a coefficient.

```python
    def _get_weights(self, X_vec, keywords_list, feature_names):
        """Put max weights for each word of redistributed mails."""
        max_ = np.max(X_vec, axis=0)
        mmax_ = np.max(max_)
        for k in keywords_list:
            if k in feature_names:
                max_[feature_names.index(k)] = mmax_ * self.keywords_coef

        return max_
```

--------------------------------

### Initialize and Use KeywordsGenerator

Source: https://melusine.readthedocs.io/en/latest/melusine.summarizer.html

Demonstrates how to initialize KeywordsGenerator, fit it with data, and transform the data to extract keywords. Assumes X and y are defined and X contains a 'tokens' column.

```python
from melusine.summarizer.keywords_generator import KeywordsGenerator
keywords_generator = KeywordsGenerator()
keywords_generator.fit(X, y)
keywords_generator.transform(X)
print(X['keywords'])
```

--------------------------------

### Print Current Name List

Source: https://melusine.readthedocs.io/en/latest/usage.html

Retrieves the configuration dictionary and prints the first 5 names from the 'names' list.

```python
conf_dict = conf.get_config_file()
print(conf_dict['words_list']['names'][:5])
```

--------------------------------

### add_boolean_answer

Source: https://melusine.readthedocs.io/en/latest/melusine.prepare_email.html

Computes a boolean Series indicating if the email header starts with a 'transfer_subject' regex. Useful for applying to DataFrame rows.

```APIDOC
## add_boolean_answer

### Description
Compute boolean Series which return True if the “header” starts with given regex ‘transfer_subject’, False if not.
To be used with methods such as: apply(func, axis=1) or apply_by_multiprocessing(func, axis=1, **kwargs).

### Parameters

- **row** (pd.Series) - Required - A row of a pandas DataFrame, expected to contain a 'header' column.

### Returns

- pd.Series - A boolean Series indicating if the header matches the transfer subject pattern.

### Examples
```python
import pandas as pd
from melusine.prepare_email.manage_transfer_reply import add_boolean_answer

# Assuming 'data' is a pandas DataFrame with a 'header' column
# data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')

# Apply to a single row
# add_boolean_answer(data.iloc[0])

# Apply to all rows
# data.apply(add_boolean_answer, axis=1)
```
```

--------------------------------

### Load Configuration from Directory

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/config/config.html

Loads configuration settings from YAML and JSON files within a specified directory, merging them into a single dictionary. It recursively searches for files and ignores notebook checkpoints.

```python
def load_conf_from_path(config_dir_path: str) -> Dict[str, Any]:
    """
    Given a directory path
    Parameters
    ----------
    config_dir_path: str
        Path to a directory containing YML or JSON conf files
    Returns
    -------
    conf: dict
        Loaded config dict
    """
    conf = dict()
    conf_files = list()
    conf_files.extend([str(f) for f in Path(config_dir_path).rglob("*.yml")])
    conf_files.extend([str(f) for f in Path(config_dir_path).rglob("*.yaml")])
    conf_files.extend([str(f) for f in Path(config_dir_path).rglob("*.json")])

    # Prevent loading notebook checkpoints
    conf_files = [x for x in conf_files if "ipynb_checkpoints" not in x]

    for name in conf_files:
        # Load YAML files
        if name.endswith(".yml") or name.endswith(".yaml"):
            logger.info(f"Loading data from file {name}")
            with open(name, "r") as f:
                tmp_conf = yaml.load(f, Loader=Loader)
                conf = update_nested_dict(conf, tmp_conf)

        # Load JSON files
        elif name.endswith(".json"):
            logger.info(f"Loading data from file {name}")
            with open(file=name, mode="r", encoding="utf-8") as f:
                tmp_conf = json.load(f)
                conf = update_nested_dict(conf, tmp_conf)

    return conf

```

--------------------------------

### Initialize Neural Network Model with Embedding

Source: https://melusine.readthedocs.io/en/latest/usage.html

Illustrates how to initialize a neural network model using a pre-trained embedding instance. The specific model definition is not provided here.

```python
# Use trained Embedding to initialise the Neural Network Model
# The definition of a neural network model is not discussed in this section
# nn_model = NeuralModel("...", pretrained_embedding=pretrained_embedding, "...")
```

--------------------------------

### Check Mail Begin by Transfer

Source: https://melusine.readthedocs.io/en/latest/melusine.prepare_email.html

Computes a boolean Series indicating if the email body starts with 'begin_transfer'. Use with pandas apply methods.

```python
import pandas as pd
from melusine.prepare_email.manage_transfer_reply import check_mail_begin_by_transfer

# Assuming 'data' is a pandas DataFrame with a 'body' column
# data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')

# Apply to a single row
# check_mail_begin_by_transfer(data.iloc[0])

# Apply to all rows
# data.apply(check_mail_begin_by_transfer, axis=1)
```

--------------------------------

### Initialize Phraser

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/nlp_tools/phraser.html

Initializes the Phraser with input and output column names and optional Gensim Phraser arguments.

```python
def __init__(self, input_column="tokens", output_column="tokens", **phraser_args):

        self.input_column = input_column
        self.output_column = output_column
        self.phraser_args = phraser_args
        self.phraser_ = None
```

--------------------------------

### Add Boolean Transfer Flag

Source: https://melusine.readthedocs.io/en/latest/melusine.prepare_email.html

Computes a boolean Series indicating if the email header starts with 'answer_subject'. Use with pandas apply methods.

```python
import pandas as pd
from melusine.prepare_email.manage_transfer_reply import add_boolean_transfer

# Assuming 'data' is a pandas DataFrame with a 'header' column
# data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')

# Apply to a single row
# add_boolean_transfer(data.iloc[0])

# Apply to all rows
# data.apply(add_boolean_transfer, axis=1)
```

--------------------------------

### Read Configuration File

Source: https://melusine.readthedocs.io/en/latest/usage.html

Reads and prints the content of the current configuration file.

```python
with open(conf.path_ini_file_, 'r') as ini_file:
    print(ini_file.read())
```

--------------------------------

### Add Boolean Answer Flag

Source: https://melusine.readthedocs.io/en/latest/melusine.prepare_email.html

Computes a boolean Series indicating if the email header starts with 'transfer_subject'. Use with pandas apply methods.

```python
import pandas as pd
from melusine.prepare_email.manage_transfer_reply import add_boolean_answer

# Assuming 'data' is a pandas DataFrame with a 'header' column
# data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')

# Apply to a single row
# add_boolean_answer(data.iloc[0])

# Apply to all rows
# data.apply(add_boolean_answer, axis=1)
```

--------------------------------

### Initialize Melusine Configuration

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/config/config.html

Initializes the global Melusine configuration object.

```python
# Load Melusine configurations
config = MelusineConfig()
```

--------------------------------

### Import Melusine Library

Source: https://melusine.readthedocs.io/en/latest/readme.html

Import the main Melusine library to begin using its functionalities.

```python
import melusine
```

--------------------------------

### Check if Email Begins by Transfer

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/manage_transfer_reply.html

Applies the `check_mail_begin_by_transfer` function to a Pandas DataFrame to identify emails that start with transfer-related content. Requires pandas and the function to be imported.

```python
import pandas as pd
from melusine.prepare_email.manage_transfer_reply import check_mail_begin_by_transfer
data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
data['is_begin_by_transfer'] = data.apply(check_mail_begin_by_transfer, axis=1)
# data contains columns ['from', 'to', 'date', 'header', 'body', 'is_begin_by_transfer']
```

--------------------------------

### Initialize and Train Neural Network

Source: https://melusine.readthedocs.io/en/latest/usage.html

Initializes a CNN model with specified text and metadata inputs, then trains it.

```python
nn_model = NeuralModel(neural_architecture_function=cnn_model,
                       pretrained_embedding=pretrained_embedding,
                       text_input_column="clean_text",
                       meta_input_list=['extension', 'dayofweek', 'hour', 'min'],
                       n_epochs=10)

nn_model.fit(X,y)
```

--------------------------------

### Email Classification with CNN Model

Source: https://melusine.readthedocs.io/en/latest/readme.html

Perform email classification using a Convolutional Neural Network (CNN) architecture. This example demonstrates training and prediction with pre-trained embeddings.

```python
from sklearn.preprocessing import LabelEncoder
from melusine.models.neural_architectures import cnn_model
from melusine.models.train import NeuralModel

X = df_email.drop(['label'], axis=1)
y = df_email.label

le = LabelEncoder()
y = le.fit_transform(y)

pretrained_embedding = embedding

nn_model = NeuralModel(architecture_function=cnn_model,
                       pretrained_embedding=pretrained_embedding,
                       text_input_column='clean_body')

nn_model.fit(X, y)
y_res = nn_model.predict(X)
```

--------------------------------

### Initialize KeywordsGenerator

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/summarizer/keywords_generator.html

Instantiate the KeywordsGenerator with various configuration options. Default values are loaded from the melusine config file.

```python
from melusine.summarizer.keywords_generator import KeywordsGenerator

keywords_generator = KeywordsGenerator(
    max_tfidf_features=10000,
    keywords=keywords,
    stopwords=stopwords,
    resample=False,
    n_jobs=1,
    progress_bar=False,
    copy=True,
    n_max_keywords=6,
    n_min_keywords=0,
    threshold_keywords=0.0,
    n_docs_in_class=100,
    keywords_coef=10,
)
```

--------------------------------

### check_mail_begin_by_transfer

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/manage_transfer_reply.html

Computes a boolean Series indicating if an email's body starts with a transfer initiation pattern. This function is designed for use with pandas DataFrame apply methods.

```APIDOC
## check_mail_begin_by_transfer

### Description
Computes a boolean Series which returns True if the email body starts with a predefined transfer initiation regex, and False otherwise. This function is intended for use with pandas DataFrame apply methods.

### Parameters
#### Path Parameters
- **row** (pd.Series) - Required - A row from a pandas DataFrame, expected to contain a 'body' column.

### Returns
- **pd.Series** - A boolean Series indicating whether the body matches the transfer initiation pattern.

### Examples
```python
import pandas as pd
data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
# Assuming 'data' DataFrame has a 'body' column
from melusine.prepare_email.manage_transfer_reply import check_mail_begin_by_transfer
# Apply to a single row
check_mail_begin_by_transfer(data.iloc[0])
# Apply to all rows in the DataFrame
data.apply(check_mail_begin_by_transfer, axis=1)
```
```

--------------------------------

### Create and Set Custom Name File

Source: https://melusine.readthedocs.io/en/latest/usage.html

Demonstrates creating a custom CSV name file and setting it as the active Melusine name file. This involves creating a pandas DataFrame, saving it to a CSV, and then updating Melusine's configuration.

```python
import pandas as pd
import os

# Create a name DataFrame
df_names = pd.DataFrame({'Name' : ['Daenerys', 'Tyrion', 'Jon', 'Raegar']})

# Path to the new name.csv file
new_path = os.path.join(os.getcwd(), 'data', 'names.csv')

# Save new name.csv file
df_names.to_csv(new_path, encoding="latin-1", sep=";", index=False)

# Set a new path to the name file in Melusine
conf.set_name_file_path(file_path=new_path)

# Print the new path to the name file
with open(conf.path_ini_file_, 'r') as ini_file:
    print(ini_file.read())

# Print the new file list
conf_dict = conf.get_config_file()
print(conf_dict['words_list']['names'][:5])
```

--------------------------------

### add_boolean_answer

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/manage_transfer_reply.html

Computes a boolean Series indicating if an email's header starts with a reply subject regex. This function is designed for use with pandas DataFrame apply methods.

```APIDOC
## add_boolean_answer

### Description
Computes a boolean Series which returns True if the email header starts with a predefined answer subject regex, and False otherwise. This function is intended for use with pandas DataFrame apply methods.

### Parameters
#### Path Parameters
- **row** (pd.Series) - Required - A row from a pandas DataFrame, expected to contain a 'header' column.

### Returns
- **pd.Series** - A boolean Series indicating whether the header matches the answer subject pattern.

### Examples
```python
import pandas as pd
data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
# Assuming 'data' DataFrame has a 'header' column
from melusine.prepare_email.manage_transfer_reply import add_boolean_answer
# Apply to a single row
add_boolean_answer(data.iloc[0])
# Apply to all rows in the DataFrame
data.apply(add_boolean_answer, axis=1)
```
```

--------------------------------

### Prepare Sequences for Neural Network

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/models/train.html

Prepares tokenized text sequences for the neural network model. Pads sequences to a fixed length using `pad_sequences`.

```python
    def _prepare_sequences(self, X):
        """Prepares the sequence to be used as input for the neural network
        model.
        The input column must be an already tokenized text : tokens
        The tokens must have been optained using the same tokenizer than the
        one used for the pre-trained embedding."""

        if isinstance(X, dict):
            seqs = [self.tokens_to_indices(X["tokens"])]
        else:
            seqs = X["tokens"].apply(self.tokens_to_indices)

        X_seq = pad_sequences(seqs, maxlen=self.seq_size)
        return X_seq
```

--------------------------------

### add_boolean_transfer

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/manage_transfer_reply.html

Computes a boolean Series indicating if an email's header starts with a transfer subject regex. This function is designed to be used with pandas DataFrame apply methods.

```APIDOC
## add_boolean_transfer

### Description
Computes a boolean Series which returns True if the email header starts with a predefined transfer subject regex, and False otherwise. This function is intended for use with pandas DataFrame apply methods.

### Parameters
#### Path Parameters
- **row** (pd.Series) - Required - A row from a pandas DataFrame, expected to contain a 'header' column.

### Returns
- **pd.Series** - A boolean Series indicating whether the header matches the transfer subject pattern.

### Examples
```python
import pandas as pd
data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
# Assuming 'data' DataFrame has a 'header' column
from melusine.prepare_email.manage_transfer_reply import add_boolean_transfer
# Apply to a single row
add_boolean_transfer(data.iloc[0])
# Apply to all rows in the DataFrame
data.apply(add_boolean_transfer, axis=1)
```
```

--------------------------------

### Save and Load Phraser Instance

Source: https://melusine.readthedocs.io/en/latest/usage.html

Demonstrates how to save a trained Phraser object to a file and then load it back. This is useful for persisting and reusing trained models.

```python
phraser.save('./pretrained_phraser.pickle')
pretrained_phraser = Phraser().load('./pretrained_phraser.pickle')
```

--------------------------------

### Get top attachment types

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/metadata_engineering.html

Calculates and returns a list of the most common attachment types from the provided data. It uses Counter to count occurrences and returns the top 'n' types.

```python
    @staticmethod
    def get_top_attachment_type(X, n=100):
        """Returns list of most common types of attachment."""

        type_counter = Counter(chain(*X["attachment_type"]))
        type_counter = type_counter.most_common(n)
        top_attachment_type = [x[0] for x in type_counter]
        return top_attachment_type
```

--------------------------------

### Initialize ConfigJsonReader

Source: https://melusine.readthedocs.io/en/latest/usage.html

Initializes the ConfigJsonReader. This is often a prerequisite for other configuration operations.

```python
import os
import pandas as pd
from melusine.config.config import ConfigJsonReader

conf = ConfigJsonReader()
```

--------------------------------

### Initialize and Use Tokenizer

Source: https://melusine.readthedocs.io/en/latest/usage.html

Shows how to initialize the Tokenizer with a specified input column and then apply it to a Pandas DataFrame to create a new 'tokens' column. The tokenizer splits sentences into sub-strings (tokens) based on NLTK's regex tokenizer.

```python
import pandas as pd
from melusine.nlp_tools.tokenizer import Tokenizer

df_tok = pd.DataFrame({
    'clean_body' : ["hello, i'm here to tokenize text. bye"],
    'clean_header' : ["re: hello"],
})

tokenizer = Tokenizer(input_column='clean_body')
df_tok = tokenizer.fit_transform(df_tok)
```

--------------------------------

### Get attachment type from row

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/metadata_engineering.html

Extracts the file extension (type) from each attachment in a row. It iterates through attachments and uses regex to find the extension, returning an empty string if an error occurs.

```python
    @staticmethod
    def get_attachment_type(row):
        """Gets type from attachment."""
        x = row["attachment"]
        attached_types = []

        try:
            for file in x:
                match = re.findall(r".*\.(.*)", file)
                if match:
                    attached_types.append(match[0])
        except Exception:
            return ""
        return attached_types
```

--------------------------------

### MelusineConfig Class Methods

Source: https://melusine.readthedocs.io/en/latest/melusine.config.html

Provides dictionary-like access and manipulation for Melusine configurations. Includes methods for copying, checking keys, and retrieving items, keys, and values.

```python
copy()


```

```python
has_key(_k_)


```

```python
items()


```

```python
keys()


```

```python
values()


```

--------------------------------

### Get Keywords for a Row

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/summarizer/keywords_generator.html

Extracts keywords from a single row of a DataFrame based on tokenized text and TF-IDF scores. It filters stopwords, digits, and applies min/max keyword count constraints.

```python
    def get_keywords(self, row):
        """Returns list of keywords in apparition order with the
        weighted tf-idf already fitted.

        Parameters
        ----------
        row : row of pd.Dataframe, columns ['tokens']

        Returns
        -------
        list of strings
        """
        tokens = self._remove_stopwords(row["tokens"])
        tokens = [x for x in tokens if not x.isdigit()]
        scores = Counter({t: self.dict_scores_.get(t, 0) for t in tokens})
        n = sum(i > self.threshold_keywords for i in list(scores.values()))
        n = min(n, self.n_max_keywords)
        n = max(n, self.n_min_keywords)
        keywords = [x[0] for x in scores.most_common(n)]
        index_sorted = [(k, tokens.index(k)) for k in keywords if k in tokens]
        index_sorted = sorted(index_sorted, key=lambda x: x[1])
        keywords_sorted = [i[0] for i in index_sorted]

        return keywords_sorted
```

--------------------------------

### Load Custom Melusine Configuration

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/config/config.html

Loads custom configuration from a specified directory if the MELUSINE_CONFIG_DIR environment variable is set. It updates the existing configuration with the custom settings.

```python
        # Load custom Melusine conf
        custom_config_directory = os.getenv("MELUSINE_CONFIG_DIR")
        if custom_config_directory:
            conf = update_nested_dict(
                conf, load_conf_from_path(custom_config_directory)
            )

        self._switch_config(conf)
```

--------------------------------

### Clone Melusine Repository

Source: https://melusine.readthedocs.io/en/latest/contributing.html

Clone your forked Melusine repository locally to begin development.

```bash
$ git clone git@github.com:your_name_here/melusine.git

```

--------------------------------

### Add Boolean Answer - melusine.prepare_email.manage_transfer_reply

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/manage_transfer_reply.html

Computes a boolean Series indicating if an email header starts with a transfer subject regex. Use with pandas `apply` for single samples or entire DataFrames.

```python
import re
from melusine import config

regex_transfer_header = config["regex"]["manage_transfer_reply"]["transfer_header"]
regex_answer_header = config["regex"]["manage_transfer_reply"]["answer_header"]
regex_begin_transfer = config["regex"]["manage_transfer_reply"]["begin_transfer"]
regex_begin_transfer_cons = config["regex"]["manage_transfer_reply"]["begin_transfer_cons"]
regex_extract_from = config["regex"]["manage_transfer_reply"]["extract_from"]
regex_extract_to = config["regex"]["manage_transfer_reply"]["extract_to"]
regex_extract_date = config["regex"]["manage_transfer_reply"]["extract_date"]
regex_extract_header = config["regex"]["manage_transfer_reply"]["extract_header"]



[docs]def add_boolean_answer(row):
    """Compute boolean Series which return True if the "header" starts with given
    regex 'transfer_subject', False if not.

    To be used with methods such as: `apply(func, axis=1)` or
    `apply_by_multiprocessing(func, axis=1, **kwargs)`.

    Parameters
    ----------
    row : row of pd.Dataframe, columns ['header']

    Returns
    -------
    pd.Series

    Examples
    --------
        >>> import pandas as pd
        >>> data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
        >>> # data contains a 'header' column

        >>> from melusine.prepare_email.manage_transfer_reply import add_boolean_answer
        >>> add_boolean_answer(data.iloc[0])  # apply for 1 sample
        >>> data.apply(add_boolean_answer, axis=1)  # apply to all samples

    """
    is_answer = False
    try:
        if re.match(regex_answer_header, row["header"]):
            is_answer = True
    except Exception:
        pass

    return is_answer

```

--------------------------------

### Prepare Data for Training

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/models/train.html

Prepares input data (X) and target labels (y) for model training. It handles categorical encoding of labels and differentiates data preparation for BERT vs. non-BERT architectures. Optionally initializes embedding matrices and models.

```python
    def _prepare_data(self, X, y, validation_data=None):
        """Prepares the data for training and validation.
        1- Encodes y to categorical
        2- Differentiates data preparation for bert models and non-bert models
        3- Creates embedding matrix and init model for training data only (validation_data=None)

        Parameters
        ----------
        X : pd.DataFrame

        y : pd.Series

        validation_data : None or tuple
            Tuple of validation data
            Data on which to evaluate the loss and any model metrics at the end of each epoch. The model will not be
            trained on this data. This could be a tuple (x_val, y_val). validation_data will override validation_split.
            Default value, None.

        Returns
        -------
        X_input : pd.DataFrame

        y_categorical : pd.Series
        """
        y_categorical = to_categorical(y)
        if not validation_data:
            nb_labels = len(np.unique(y))

        if self.architecture_function.__name__ != "bert_model":
            X = self.tokenizer.transform(X)
            X_meta, nb_meta_features = self._get_meta(X)

            if not validation_data:
                if self.pretrained_embedding:
                    self._get_embedding_matrix()
                else:
                    self._create_vocabulary_from_tokens(X)
                    self._generate_random_embedding_matrix()
                self.vocabulary_dict = {
                    word: i for i, word in enumerate(self.vocabulary) 
                }
                if self.architecture_function.__name__ == "flipout_cnn_model":
                    # this variational model needs also the size of the training dataset
                    training_data_size = len(X)
                    self.model = self.architecture_function(
                        embedding_matrix_init=self.embedding_matrix,
                        ntargets=nb_labels,
                        seq_max=self.seq_size,
                        nb_meta=nb_meta_features,
                        loss=self.loss,
                        activation=self.activation,
                        training_data_size=training_data_size
                    )
                else:
                    self.model = self.architecture_function(
                        embedding_matrix_init=self.embedding_matrix,
                        ntargets=nb_labels,
                        seq_max=self.seq_size,
                        nb_meta=nb_meta_features,
                        loss=self.loss,
                        activation=self.activation,
                    )

            X_seq = self._prepare_sequences(X)

            if nb_meta_features == 0:
                X_input = X_seq
            else:
                X_input = [X_seq, X_meta]

        else:
            X_seq, X_attention = self._prepare_bert_sequences(X)
            X_meta, nb_meta_features = self._get_meta(X)
            self.nb_labels, self.nb_meta_features = nb_labels, nb_meta_features
            if not validation_data:
                self.model = self.architecture_function(
                    ntargets=nb_labels,
                    seq_max=self.seq_size,
                    nb_meta=nb_meta_features,
                    loss=self.loss,
                    activation=self.activation,
                    bert_model=self.bert_model,
                )

            if nb_meta_features == 0:
                X_input = [X_seq, X_attention]
            else:
                X_input = [X_seq, X_attention, X_meta]

        return X_input, y_categorical
```

--------------------------------

### Add Boolean Transfer - melusine.prepare_email.manage_transfer_reply

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/prepare_email/manage_transfer_reply.html

Computes a boolean Series indicating if an email header starts with a transfer subject regex. Use with pandas `apply` for single samples or entire DataFrames.

```python
import re
from melusine import config

regex_transfer_header = config["regex"]["manage_transfer_reply"]["transfer_header"]
regex_answer_header = config["regex"]["manage_transfer_reply"]["answer_header"]
regex_begin_transfer = config["regex"]["manage_transfer_reply"]["begin_transfer"]
regex_begin_transfer_cons = config["regex"]["manage_transfer_reply"]["begin_transfer_cons"]
regex_extract_from = config["regex"]["manage_transfer_reply"]["extract_from"]
regex_extract_to = config["regex"]["manage_transfer_reply"]["extract_to"]
regex_extract_date = config["regex"]["manage_transfer_reply"]["extract_date"]
regex_extract_header = config["regex"]["manage_transfer_reply"]["extract_header"]



[docs]def add_boolean_transfer(row):
    """Compute boolean Series which return True if the "header" starts with given
    regex 'answer_subject', False if not.

    To be used with methods such as: `apply(func, axis=1)` or
    `apply_by_multiprocessing(func, axis=1, **kwargs)`.

    Parameters
    ----------
    row : row of pd.Dataframe, columns ['header']

    Returns
    -------
    pd.Series

    Examples
    --------
        >>> import pandas as pd
        >>> data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
        >>> # data contains a 'header' column

        >>> from melusine.prepare_email.manage_transfer_reply import add_boolean_transfer
        >>> add_boolean_transfer(data.iloc[0])  # apply for 1 sample
        >>> data.apply(add_boolean_transfer, axis=1)  # apply to all samples

    """
    is_transfer = False
    try:
        if re.match(regex_transfer_header, row["header"]):
            is_transfer = True
    except Exception:
        pass

    return is_transfer

```

--------------------------------

### Create Vocabulary from Tokens

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/models/train.html

Creates a word index dictionary from a series of tokens. Used to build the model's vocabulary based on token frequency.

```python
    def _create_vocabulary_from_tokens(self, X):
        """Create a word indexes dictionary from tokens."""
        token_series = X["tokens"]
        c = Counter([token for token_list in token_series for token in token_list])
        self.vocabulary = [t[0] for t in c.most_common(self.vocab_size)]
        pass
```

--------------------------------

### Initialize and Train NeuralModel

Source: https://melusine.readthedocs.io/en/latest/melusine.models.html

Initializes a `NeuralModel` with a specified architecture, pre-trained embeddings, and metadata columns, then fits the model to training data. Ensure `X_train` and `y_train` are prepared pandas DataFrame and Series respectively. `pretrained_embedding` should be loaded using `Embedding.load()`.

```python
from melusine.models.train import NeuralModel
from melusine.models.neural_architectures import cnn_model
from melusine.nlp_tools.embedding import Embedding
pretrained_embedding = Embedding.load()
list_meta = ['extension', 'dayofweek', 'hour']
nn_model = NeuralModel(cnn_model, pretrained_embedding, list_meta)  #noqa
nn_model.fit(X_train, y_train)  #noqa
y_res = nn_model.predict(X_test)  #noqa
```

--------------------------------

### Train and Load Embedding

Source: https://melusine.readthedocs.io/en/latest/melusine.nlp_tools.html

Demonstrates how to train an embedding model, save it to a file, and then load it back. This is useful for reusing trained embeddings.

```python
from melusine.nlp_tools.embedding import Embedding
embedding = Embedding()
embedding.train(X)  # noqa
embedding.save(filepath)  # noqa
embedding = Embedding().load(filepath)  # noqa
```

--------------------------------

### Initialize RegexTokenizer

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/nlp_tools/tokenizer.html

Initializes the RegexTokenizer with a custom regex pattern or uses the default. Handles stopwords by creating a set.

```python
tokenizer_regex: str = r"\w+(?:[?\-_']\w+)*"
stopwords: List[str] = None
```

--------------------------------

### Prepare Email Data for Segmentation

Source: https://melusine.readthedocs.io/en/latest/melusine.prepare_email.html

This snippet demonstrates how to load email data and prepare it by building a structured historic column, which is a prerequisite for using email segmentation functions.

```python
import pandas as pd
from melusine.prepare_email.build_historic import build_historic
data = pd.read_pickle('./tutorial/data/emails_anonymized.pickle')
data['structured_historic'] = data.apply(build_historic, axis=1)
# data contains column ['structured_historic']
```

--------------------------------

### MelusineConfig Class Initialization

Source: https://melusine.readthedocs.io/en/latest/_modules/melusine/config/config.html

Initializes the MelusineConfig class, loading default configurations and allowing them to be overridden by custom configurations specified via the MELUSINE_CONFIG_DIR environment variable.

```python
class MelusineConfig:
    """
    The MelusineConfig class acts as a dict containing configurations.
    The configurations can be changed dynamically using the switch_config function.
    """

    def __init__(self):
        super().__init__()
        self._config = None
        self.load_melusine_conf()

    def __getitem__(self, key):
        """
        Access configuration elements
        """
        return self._config[key]

    def __repr__(self):
        """
        Represent the MelusineConfig class
        """
        return repr(self._config)

    def __len__(self):
        """
        Returns the length of the config dict
        """
        return len(self._config)


    def copy(self):
        """
        Copy the config dict
        """
        return self._config.copy()



    def has_key(self, k):
        """
        Checks if given key exists in the config dict
        """
        return k in self._config



    def keys(self):
        """
        Returns the keys of the config dict
        """
        return self._config.keys()



    def values(self):
        """
        Returns the values of the config dict
        """
        return self._config.values()



    def items(self):
        """
        Returns the items of the config dict
        """
        return self._config.items()


    def __contains__(self, item):
        """
        Checks if the given item is contained in the config dict
        """
        return item in self._config

    def __iter__(self):
        """
        Iterates over the the config dict
        """
        return iter(self._config)


    def load_melusine_conf(self) -> None:
        """
        Load the melusine configurations.
        The default configurations are loaded first (the one present in the melusine package).
        Custom configurations may overwrite the default ones.
        Custom configuration should be specified in YML and JSON files and placed in a directory.
        The directory path should be set as the value of the MELUSINE_CONFIG_DIR environment variable.
        Returns
        -------
        conf: dict
            Loaded config dict
        """
        conf = dict()

        # Load default Melusine conf
        default_config_directory = op.dirname(op.abspath(__file__))
        conf = update_nested_dict(conf, load_conf_from_path(default_config_directory))


```

--------------------------------

### Load Melusine Configurations

Source: https://melusine.readthedocs.io/en/latest/melusine.config.html

Loads Melusine configurations from default files and custom YML/JSON files specified by the MELUSINE_CONFIG_DIR environment variable. Custom configurations can overwrite default ones.

```python
load_melusine_conf() → None
    
    Load the melusine configurations. The default configurations are loaded first (the one present in the melusine package). Custom configurations may overwrite the default ones. Custom configuration should be specified in YML and JSON files and placed in a directory. The directory path should be set as the value of the MELUSINE_CONFIG_DIR environment variable. Returns ——- conf: dict
> Loaded config dict 

```

--------------------------------

### Create a New Branch for Development

Source: https://melusine.readthedocs.io/en/latest/contributing.html

Create a new branch for your bug fix or feature development.

```bash
$ git checkout -b name-of-your-bugfix-or-feature

```

--------------------------------

### Reset Configuration Path

Source: https://melusine.readthedocs.io/en/latest/usage.html

Resets the configuration path to the original default.

```python
from melusine.config.config import ConfigJsonReader

conf = ConfigJsonReader()
conf.reset_config_path()
```

--------------------------------

### Deploy Melusine with Bumpversion

Source: https://melusine.readthedocs.io/en/latest/contributing.html

Deploy a new version of Melusine by committing changes, updating the history, and using bumpversion to increment the version number.

```bash
$ bumpversion patch # possible: major / minor / patch
$ git push
$ git push --tags

```

--------------------------------

### Full Email Preprocessing Pipeline

Source: https://melusine.readthedocs.io/en/latest/usage.html

Assemble a complete preprocessing pipeline using Pipeline, chaining multiple TransformerScheduler objects and other transformers.

```python
PreprocessingPipeline = Pipeline([
    ('ManageTransferReply', ManageTransferReply),
    ('Segmenting', Segmenting),
    ('LastBodyHeaderCleaning', LastBodyHeaderCleaning),
    ('PhraserTransformer', PhraserTransformer),
    ('tokenizer', tokenizer)
])
```

--------------------------------

### Create and Chain Melusine Transformers

Source: https://melusine.readthedocs.io/en/latest/usage.html

Demonstrates creating a transformer with regular Melusine functions and chaining it with a custom transformer in a pipeline. This pipeline is then used to pre-process an input DataFrame.

```python
from melusine.prepare_email.manage_transfer_reply import add_boolean_answer, add_boolean_transfer
from melusine.transformers import TransformerScheduler, Pipeline
from melusine.transformers.count_word_occurrence import CountWordOccurrence

# Create a second transformer with regular Melusine functions
ManageTransferReply = TransformerScheduler(
    functions_scheduler=[
        (add_boolean_answer, None, ['is_answer']),
        (add_boolean_transfer, None, ['is_transfer'])
    ])

# Chain transformers in a pipeline
prepare_data_pipeline = Pipeline([
    ('CountWordOccurrence', CountWordOccurrence), # Transformer with custom functions
    ('ManageTransferReply', ManageTransferReply), # Transformer with regular Melusine functions
])

# Pre-process input DataFrame
df_duck_prep = prepare_data_pipeline.fit_transform(df_duck)
```

--------------------------------

### Configuration Utility Functions

Source: https://melusine.readthedocs.io/en/latest/melusine.config.html

Utility functions for managing Melusine configurations, including loading from paths, updating dictionaries, and handling deprecation warnings.

```APIDOC
## Utility Functions

### `melusine.config.config.config_deprecation_warnings(_config_dict : Dict[str, Any]_)`

#### Description
Raise Deprecation Warning when using deprecated configs.

### `melusine.config.config.load_conf_from_path(_config_dir_path : str_)`

#### Description
Given a directory path, load configurations from YML or JSON files.

#### Parameters
- `config_dir_path` (str): Path to a directory containing YML or JSON conf files.

#### Returns
- `conf` (dict): Loaded config dict.

### `melusine.config.config.switch_config(_new_config_)`

#### Description
Function to change the Melusine configuration using a dict.

#### Parameters
- `new_config` (dict): Dict containing the new config.

### `melusine.config.config.update_nested_dict(_base_dict : dict_, _update_dict : dict_)`

#### Description
Update a (possibly) nested dictionary using another (possibly) nested dictionary.

#### Parameters
- `base_dict` (Mapping): Base dict to be updated.
- `update_dict` (Mapping): Update dict to merge into `base_dict`.

#### Returns
- `base_dict` (Mapping): Updated dict.
```