scrubadub

https://scrubadub.readthedocs.io/en/stable/
Admin
scrubadub is a Python package that removes personally identifiable information such as names, email...
Tokens:72,628
Snippets:453
Trust Score:9.6
Update:6 months ago
Show doc for...
Tokens:
Raw
### Setup development environment

Source: https://scrubadub.readthedocs.io/en/stable/contributing.html

Commands to create a virtual environment and install necessary development dependencies.

```bash
mkvirtualenv scrubadub
pip install -r requirements/python-dev
```

--------------------------------

### Install libpostal on Linux

Source: https://scrubadub.readthedocs.io/en/stable/addresses.html

Follow these commands to install the libpostal library on a Linux system. Ensure you have curl, autoconf, automake, libtool, and pkg-config installed.

```bash
$ sudo apt-get install curl autoconf automake libtool pkg-config
$ git clone https://github.com/openvenues/libpostal
$ cd libpostal
$ ./bootstrap.sh
$ ./configure --prefix=/usr/local/
$ make -j4
$ sudo make install

```

--------------------------------

### Run the test suite

Source: https://scrubadub.readthedocs.io/en/stable/contributing.html

Execute the project's test suite to verify the local installation.

```bash
./tests/run.py
```

--------------------------------

### Install Scrubadub Base Package

Source: https://scrubadub.readthedocs.io/en/stable/names.html

Install the base scrubadub package, which includes the TextBlob detector.

```bash
$ pip install scrubadub

```

--------------------------------

### Install Scrubadub Stanford Detector

Source: https://scrubadub.readthedocs.io/en/stable/names.html

Install the necessary Python package for the Stanford NER detector.

```bash
$ pip install scrubadub_stanford

```

--------------------------------

### Registering a custom PostProcessor

Source: https://scrubadub.readthedocs.io/en/stable/api_scrubadub_post.html

Provides an example of how to register a custom PostProcessor with the Scrubber class, allowing it to be automatically loaded.

```python
scrubadub.post_processors.register_post_processor(_post_processor : Type[PostProcessor]_, _autoload : Optional[bool] = None_, _index : Optional[int] = None_) -> None:
    """
    Register a PostProcessor for use with the `Scrubber` class.
    You can use `register_post_processor(NewPostProcessor)` after your post-processor definition to automatically register it with the `Scrubber` class so that it can be used to process Filth.
    The argument `autoload` sets if a new `Scrubber()` instance should load this `PostProcessor` by default. 

    Parameters
        
      * **post_processor** (_PostProcessor class_) – The `PostProcessor` to register with the scrubadub post-processor configuration.
      * **autoload** (_bool_) – Whether to automatically load this `Detector` on `Scrubber` initialisation.
      * **index** (_int_) – The location/index in which this `PostProcessor` should be added.
    """
```

--------------------------------

### Install scrubadub_address Python Package

Source: https://scrubadub.readthedocs.io/en/stable/addresses.html

Install the scrubadub_address package using pip after libpostal has been successfully installed. This will also install pypostal and pyap as dependencies.

```bash
$ pip install scrubadub_address

```

--------------------------------

### Install scrubadub via pip

Source: https://scrubadub.readthedocs.io/en/stable

Command to install the scrubadub package using the pip package manager.

```bash
pip install scrubadub
```

--------------------------------

### Install Java for Stanford NER

Source: https://scrubadub.readthedocs.io/en/stable/names.html

Install Java 14 JRE on Debian Linux systems to use the Stanford NER detector.

```bash
$ apt-get install openjdk-14-jre

```

--------------------------------

### Install Scrubadub Spacy Detector

Source: https://scrubadub.readthedocs.io/en/stable/names.html

Install the Python package for the Spacy detector. Spacy v3 requires Python 3.6 to 3.8.

```bash
$ pip install scrubadub_spacy

```

--------------------------------

### UserSuppliedFilthDetector Example

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/user_supplied.html

Demonstrates how to use the UserSuppliedFilthDetector to replace known names in a text. This detector requires a list of dictionaries, each specifying a 'match' string and a 'filth_type'.

```python
>>> import scrubadub
>>> scrubber = scrubadub.Scrubber(detector_list=[
...     scrubadub.detectors.UserSuppliedFilthDetector([
...         {'match': 'Anika', 'filth_type': 'name'},
...         {'match': 'Larry', 'filth_type': 'name'},
...     ]),
... ])
>>> scrubber.clean("Anika is my favourite employee.")
'{{NAME}} is my favourite employee.'
```

--------------------------------

### Create a Custom Filth Detector

Source: https://scrubadub.readthedocs.io/en/stable/usage.html

Extend Scrubadub by creating custom Filth types and Detector classes. The example shows how to define a new Filth type 'mine' and a detector 'MyDetector' that yields this custom filth.

```python
import scrubadub

class MyFilth(scrubadub.filth.Filth):
    type = 'mine'

class MyDetector(scrubadub.detectors.Detector):
    name = 'my_detector'
    def iter_filth(self, text, document_name=None):
        # This detector always returns this same Filth no matter the input.
        # You should implement something better here.
        yield MyFilth(beg=0, end=8, text='My stuff', document_name=document_name, detector_name=self.name)

scrubber = scrubadub.Scrubber()
scrubber.add_detector(MyDetector)
text = "My stuff can be found there."
scrubber.clean(text)
```

--------------------------------

### Implement a Custom Post-Processor

Source: https://scrubadub.readthedocs.io/en/stable/usage.html

Add custom post-processing logic to Scrubadub by creating a PostProcessor subclass. The PIIReplacer example modifies the 'replacement_string' for detected filth to 'PII'.

```python
import scrubadub

class PIIReplacer(scrubadub.post_processors.PostProcessor):
    name = 'pii_replacer'
    def process_filth(self, filth_list):
        for filth in filth_list:
            # replacement_string is what the Filth will be replaced by
            filth.replacement_string = 'PII'
        return filth_list

scrubber = scrubadub.Scrubber(post_processor_list=[
    PIIReplacer(),
    scrubadub.post_processors.PrefixSuffixReplacer(),
])
text = "contact me on (478)345-1309 or joe@example.com"
scrubber.clean(text)
```

--------------------------------

### Use Stanford NER Detector

Source: https://scrubadub.readthedocs.io/en/stable/names.html

Add and use the StanfordEntityDetector with a Scrubber instance. Requires `scrubadub_stanford` to be installed.

```python
>>> import scrubadub, scrubadub_stanford
>>> scrubber = scrubadub.Scrubber()
>>> scrubber.add_detector(scrubadub_stanford.detectors.StanfordEntityDetector)
>>> scrubber.clean("My name is John")
'My name is {{NAME}}'

```

--------------------------------

### Define a Custom Detector

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/base.html

Example of creating a new detector by subclassing Detector and implementing iter_filth. Requires defining a custom Filth class.

```python
>>> import scrubadub
>>> class MyFilth(scrubadub.filth.Filth):
...     type = 'mine'
>>> class MyDetector(scrubadub.detectors.Detector):
...     name = 'my_fr_detector'
...     def iter_filth(self, text, document_name=None):
...         # This detector always returns this same Filth no matter the input.
...         # You should implement something better here.
...         yield MyFilth(beg=0, end=8, text='My stuff', document_name=document_name, detector_name=self.name)
>>> scrubber = scrubadub.Scrubber()
>>> scrubber.add_detector(MyDetector)
>>> text = "My stuff can be found there."
>>> scrubber.clean(text)
'{{MINE}} can be found there.'
```

--------------------------------

### Configure and Use SpacyNameDetector for German

Source: https://scrubadub.readthedocs.io/en/stable/api_scrubadub_detectors.html

This example demonstrates how to configure the SpacyNameDetector for the German language by setting custom noun tags and name prefixes. It then uses the configured detector with a Scrubber to clean a German text.

```python
import scrubadub, scrubadub_spacy
scrubadub_spacy.detectors.spacy_name_title.SpacyNameDetector.NOUN_TAGS['de'] = ['NN', 'NE', 'NNE']
scrubadub_spacy.detectors.spacy_name_title.SpacyNameDetector.NAME_PREFIXES['de'] = ['frau', 'herr']
detector = scrubadub_spacy.detectors.spacy_name_title.SpacyNameDetector(locale='de_DE', model='de_core_news_sm',
    include_spacy=False)
scrubber = scrubadub.Scrubber(detector_list=[detector], locale='de_DE')
scrubber.clean("bleib dort Frau Schmidt")
```

--------------------------------

### Customize filth markers with hash and HTML

Source: https://scrubadub.readthedocs.io/en/stable/usage.html

Customize filth markers to include a hash of the filth and use custom HTML tags for replacement. This example uses `FilthReplacer` with hash options and `PrefixSuffixReplacer` for bold HTML tags.

```python
>>> import scrubadub
>>> scrubber = scrubadub.Scrubber(post_processor_list=[
...     scrubadub.post_processors.FilthReplacer(include_hash=True, hash_salt='example', hash_length=5),
...     scrubadub.post_processors.PrefixSuffixReplacer(prefix='<b>', suffix='</b>'),
... ])
>>> scrubber.clean("contact me on (478)345-1309 or joe@example.com")
'contact me on <b>PHONE-DB92D</b> or <b>EMAIL-028CC</b>'
```

--------------------------------

### Filth Class Initialization

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/filth/base.html

Initializes a Filth object with optional parameters for its position, text, detector, and document. Raises a ValueError if the start position is greater than or equal to the end position.

```python
def __init__(self, beg: Optional[int] = None, end: Optional[int] = None, text: Optional[str] = None,
                 match: Optional[Match] = None, detector_name: Optional[str] = None,
                 document_name: Optional[str] = None, replacement_string: Optional[str] = None,
                 locale: Optional[str] = None, **kwargs):

        self.beg = 0  # type: int
        self.end = 0  # type: int
        self.text = ''  # type: str
        self.match = None  # type: Optional[Match]

        if match is not None and isinstance(match, Match):
            self.beg = match.start()
            self.end = match.end()
            self.text = match.string[match.start():match.end()]
            self.match = match

        if beg is not None:
            self.beg = beg
        if end is not None:
            self.end = end
        if text is not None:
            self.text = text

        self.detector_name = detector_name  # type: Optional[str]
        self.document_name = document_name  # type: Optional[str]
        self.replacement_string = replacement_string  # type: Optional[str]
        self.locale = locale  # type: Optional[str]

        if self.beg >= self.end:
            raise ValueError(
                f"Creating invalid filth (self.beg >= self.end): {self}"
            )
```

--------------------------------

### Check and Download spaCy Model

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub_spacy/detectors/spacy.html

Verifies if a specified spaCy model is installed. If not, it logs a message and attempts to download it using `spacy.cli.download`. Returns true if the model is available.

```python
spacy_info = spacy.info()
        if isinstance(spacy_info, str):
            raise ValueError('Unable to detect spacy models.')
        models = list(spacy_info.get('pipelines', spacy_info.get('models', None)).keys())
        if models is None:
            raise ValueError('Unable to detect spacy models.')

        if model not in models:
            logger = logging.getLogger('scrubadub.detectors.spacy.SpacyEntityDetector')
            logger.info("Downloading spacy model {}".format(model))
            spacy.cli.download(model)
            importlib.import_module(model)
            # spacy.info() doesnt update after a spacy.cli.download, so theres no point checking it
            models.append(model)

        # Always returns true, if it fails to download, spacy sys.exit()s
        return model in models
```

--------------------------------

### Configure SpacyEntityDetector with a specific model

Source: https://scrubadub.readthedocs.io/en/stable/usage.html

Use the `Scrubber` and `SpacyEntityDetector` to identify named entities in a specific language. Ensure the `scrubadub_spacy` library is installed and the specified model is available.

```python
>>> import scrubadub, scrubadub_spacy
>>> scrubber = scrubadub.Scrubber(locale='fr_FR')
>>> detector = scrubadub_spacy.detectors.SpacyEntityDetector(model='fr_core_news_lg')
>>> scrubber.add_detector(detector)
>>> text = "contacter Emmanuel Pereira au 01 81 36 78 86"
>>> scrubber.clean(text)
'contacter {{NAME}} au {{PHONE}}'
```

--------------------------------

### Registering a detector via entry points

Source: https://scrubadub.readthedocs.io/en/stable/creating_detectors.html

Use entry points in setup.cfg to register detectors from separate packages that are not yet imported.

```ini
[options.entry_points]
scrubadub_detectors =
    orange = scrubadub_fruit.detectors:OrangeDetector
```

--------------------------------

### Clone the repository

Source: https://scrubadub.readthedocs.io/en/stable/contributing.html

Initial step to obtain a local copy of the project source code.

```bash
git clone https://github.com/YOUR-USERNAME/scrubadub.git
```

--------------------------------

### Scrubber with FilthReplacer (Basic)

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/post_processors/filth_replacer.html

Demonstrates using the FilthReplacer post-processor with default settings to replace detected filth with their types (e.g., PHONE, EMAIL).

```python
>>> import scrubadub
>>> scrubber = scrubadub.Scrubber(post_processor_list=[
...     scrubadub.post_processors.FilthReplacer(),
... ])
>>> scrubber.clean("Contact me at 522-368-8530 or hernandezjenna@example.com")
'Contact me at PHONE or EMAIL'
```

--------------------------------

### Configure and use DateOfBirthDetector

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/date_of_birth.html

Demonstrates how to customize the minimum age for detection and initialize the scrubber with the detector.

```python
>>> import scrubadub, scrubadub.detectors.date_of_birth
>>> DateOfBirthFilth.min_age_years = 12
>>> scrubber = scrubadub.Scrubber(detector_list=[
...     scrubadub.detectors.date_of_birth.DateOfBirthDetector(),
... ])
>>> scrubber.clean("I was born on 10-Nov-2008.")
'I was born {{DATE_OF_BIRTH}}.'
```

--------------------------------

### Configure and use SpacyEntityDetector

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub_spacy/detectors/spacy.html

Demonstrates how to define a custom Filth class, map it to a spaCy entity label, and use the detector within a Scrubber instance.

```python
>>> import scrubadub, scrubadub_spacy
>>> class MoneyFilth(scrubadub.filth.Filth):
...     type = 'money'
>>> scrubadub_spacy.detectors.spacy.SpacyEntityDetector.filth_cls_map['MONEY'] = MoneyFilth
>>> detector = scrubadub_spacy.detectors.spacy.SpacyEntityDetector(named_entities=['MONEY'])
>>> scrubber = scrubadub.Scrubber(detector_list=[detector])
>>> scrubber.clean("You owe me 12 dollars man!")
'You owe me {{MONEY}} man!'
```

--------------------------------

### GET /detectors/date_of_birth/supported_locale

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/date_of_birth.html

Checks if the DateOfBirthDetector supports a specific locale.

```APIDOC
## GET /detectors/date_of_birth/supported_locale

### Description
Returns true if the DateOfBirthDetector supports the given locale.

### Method
GET

### Endpoint
/detectors/date_of_birth/supported_locale

### Parameters
#### Query Parameters
- **locale** (string) - Required - The locale of the documents in the format: 2 letter lower-case language code eg "en", "es".

### Response
#### Success Response (200)
- **supported** (boolean) - True if the locale is supported, otherwise False.
```

--------------------------------

### Add Prefix and Suffix to Filth Replacement

Source: https://scrubadub.readthedocs.io/en/stable/api_scrubadub_post.html

Demonstrates initializing a Scrubber with a FilthReplacer, implying that subsequent replacements might be affected by default prefix/suffix settings if PrefixSuffixReplacer were also included or if FilthReplacer had such options.

```python
>>> import scrubadub
>>> scrubber = scrubadub.Scrubber(post_processor_list=[
...     scrubadub.post_processors.FilthReplacer(),
... ])
```

--------------------------------

### Use Spacy NER Detector

Source: https://scrubadub.readthedocs.io/en/stable/names.html

Add and use the SpacyEntityDetector with a Scrubber instance. Requires `scrubadub_spacy` to be installed.

```python
>>> import scrubadub, scrubadub_spacy
>>> scrubber = scrubadub.Scrubber()
>>> scrubber.add_detector(scrubadub_spacy.detectors.SpacyEntityDetector)
>>> scrubber.clean("My name is John")
'My name is {{NAME}}'

```

--------------------------------

### FilthReplacer Initialization

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/post_processors/filth_replacer.html

Initializes the FilthReplacer post-processor with configuration options for how filth should be replaced in the text.

```APIDOC
## FilthReplacer.__init__

### Description
Initializes the FilthReplacer instance with specific formatting rules for replacement tokens.

### Parameters
#### Request Body
- **include_type** (bool) - Optional - Whether to include the filth type in the replacement label. Default: True.
- **include_count** (bool) - Optional - Whether to include a unique count for each piece of filth. Default: False.
- **include_hash** (bool) - Optional - Whether to include a hash of the filth content. Default: False.
- **uppercase** (bool) - Optional - Whether to make the label uppercase. Default: True.
- **separator** (str) - Optional - Separator used for merged filth labels. Default: '+'.
- **hash_length** (int) - Optional - Length of the hexadecimal hash. Default: 16.
- **hash_salt** (str/bytes) - Optional - Salt used for the hashing process.
```

--------------------------------

### Scrubber with FilthReplacer (Include Hash)

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/post_processors/filth_replacer.html

Demonstrates using the FilthReplacer post-processor with hashing enabled. It includes the filth type and an 8-character hash of the filth, using a specified salt.

```python
>>> scrubber = scrubadub.Scrubber(post_processor_list=[
...     scrubadub.post_processors.FilthReplacer(include_hash=True, hash_salt='example', hash_length=8),
... ])
>>> scrubber.clean("Contact me at 522-368-8530 or hernandezjenna@example.com")
'Contact me at PHONE-7358BF44 or EMAIL-AC0B8AC3'
```

--------------------------------

### Check spaCy Version

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub_spacy/detectors/spacy.html

Ensures that the installed spaCy version is v3. Raises an ImportError if the version is incorrect or cannot be detected.

```python
spacy_version = spacy.__version__  # spacy_info.get('spaCy version', spacy_info.get('spacy_version', None))

        if spacy_version is None:
            raise ImportError('Spacy v3 needs to be installed. Unable to detect spacy version.')
try:
            spacy_major = int(spacy_version.split('.')[0])
        except Exception:
            raise ImportError('Spacy v3 needs to be installed. Spacy version {} is unknown.'.format(spacy_version))
        if spacy_major != 3:
            raise ImportError('Spacy v3 needs to be installed. Detected version {}.'.format(spacy_version))

        return True
```

--------------------------------

### Scrub Document and Generate Classification Report

Source: https://scrubadub.readthedocs.io/en/stable/accuracy.html

Initializes a Scrubber, adds a TaggedEvaluationFilthDetector with known PII, iterates through the document to find filth, and then generates a classification report. This demonstrates how to measure detector efficiency.

```python
scrubber = scrubadub.Scrubber()
scrubber.add_detector(scrubadub.detectors.TaggedEvaluationFilthDetector(known_filth_items=tagged_pii))
filth_list = list(scrubber.iter_filth(document))
print(scrubadub.comparison.get_filth_classification_report(filth_list))
```

--------------------------------

### Get Counts from FilthTypePositions

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/comparison.html

Merges positions and then calculates the counts of detected and tagged filth items, returning the results as a pandas DataFrame.

```python
def get_counts(self) -> pd.DataFrame:
        self.merge_positions()

        data_list = []  # type: List[Dict[Tuple[str, ...], int]]
        for position in self.positions:
            row = {
                detected_name: 1
                for detected_name in position.detected
            }
            row.update({
                detected_name: 1
                for detected_name in position.tagged
            })
            data_list.append(row)

        dataframe = pd.DataFrame(data_list).fillna(0).astype(int)
        dataframe.columns = pd.MultiIndex.from_tuples(
            dataframe.columns.values.tolist(),
            names=self.column_names,
        )

        return dataframe
```

--------------------------------

### Add prefix and suffix to replacements

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/post_processors/prefix_suffix.html

Use PrefixSuffixReplacer to wrap replacement strings with specified prefixes and suffixes. This example demonstrates wrapping with '{{' and '}}'.

```python
import scrubadub
scrubber = scrubadub.Scrubber(post_processor_list=[
    scrubadub.post_processors.FilthReplacer(),
    scrubadub.post_processors.PrefixSuffixReplacer(prefix='{{', suffix='}}'),
])
scrubber.clean("Contact me at 522-368-8530 or hernandezjenna@example.com")
```

--------------------------------

### Initialise FilthReplacer Post-Processor

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/post_processors/filth_replacer.html

Initialises the FilthReplacer with options to include type, count, hash, case, separator, and hashing parameters. The hash salt is encoded to bytes if provided as a string, otherwise a random salt is generated.

```python
def __init__(self, include_type: bool = True, include_count: bool = False, include_hash: bool = False,
                 uppercase: bool = True, separator: Optional[str] = None, hash_length: Optional[int] = None,
                 hash_salt: Optional[Union[str, bytes]] = None, **kwargs):
        """Initialise the FilthReplacer.

        :param include_type:
        :type include_type: bool, default True
        :param include_count:
        :type include_count: bool, default False
        :param include_hash:
        :type include_hash: bool, default False
        :param uppercase: Make the label uppercase
        :type uppercase: bool, default True
        :param separator: Used to separate labels if a merged filth is being replaced
        :type separator: Optional[str], default None
        :param hash_length: The length of the hexadecimal hash
        :type hash_length: Optional[int], default None
        :param hash_salt: The salt used in the hashing process
        :type hash_salt: Optional[Union[str, bytes]], default None
        """
        super(FilthReplacer, self).__init__(**kwargs)
        self.include_type = include_type
        self.include_count = include_count
        self.include_hash = include_hash
        self.uppercase = uppercase
        self.separator = separator or '+'
        self.hash_length = hash_length or 16

        if isinstance(hash_salt, str):
            self.hash_salt = hash_salt.encode('utf8')  # type: bytes
        else:
            self.hash_salt = os.urandom(128)
```

--------------------------------

### Scrubber with FilthReplacer (Include Count)

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/post_processors/filth_replacer.html

Demonstrates using the FilthReplacer post-processor with the count option enabled. This replaces identical filth instances with the same sequential number.

```python
>>> scrubber = scrubadub.Scrubber(post_processor_list=[
...     scrubadub.post_processors.FilthReplacer(include_count=True),
... ])
>>> scrubber.clean("Contact me at taylordaniel@example.com or hernandezjenna@example.com, "
...                "but taylordaniel@example.com is probably better.")
'Contact me at EMAIL-0 or EMAIL-1, but EMAIL-0 is probably better.'
```

--------------------------------

### Sort Filth Objects

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/scrubbers.html

Sorts a sequence of Filth objects by document name, start position, and length. This is a prerequisite for merging and concatenation operations.

```python
@staticmethod
def _sort_filths(filth_list: Sequence[Filth]) -> List[Filth]:
    """Sorts a list of filths, needed before merging and concatenating"""
    # Sort by start position. If two filths start in the same place then
    # return the longer one first
    filth_list = list(filth_list)
    filth_list.sort(key=lambda f: (
        str(getattr(f, 'document_name', None) if hasattr(f, 'document_name') else ''), f.beg, -f.end
    ))
    return filth_list
```

--------------------------------

### Initialize Scrubber with Custom Detectors and Post-Processors

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/scrubbers.html

Initializes a Scrubber instance, optionally specifying custom lists of detectors and post-processors. If not provided, it defaults to loading all autoloading detectors and post-processors that support the specified locale.

```python
import warnings
from typing import Optional, Sequence, Generator, Dict, Type, Union, List

from . import detectors
from . import post_processors
from .detectors import Detector
from .post_processors import PostProcessor
from .filth import Filth



[docs]class Scrubber:
    """The Scrubber class is used to clean personal information out of dirty
    dirty text. It manages a set of ``Detector``'s that are each responsible
    for identifying ``Filth``. ``PostProcessor`` objects are used to alter
    the found Filth. This could be to replace the Filth with a hash or token.
    """


[docs]    def __init__(self, detector_list: Optional[Sequence[Union[Type[Detector], Detector, str]]] = None,
                 post_processor_list: Optional[Sequence[Union[Type[PostProcessor], PostProcessor, str]]] = None,
                 locale: Optional[str] = None):
        """Create a ``Scrubber`` object.

        :param detector_list: The list of detectors to use in this scrubber.
        :type detector_list: Optional[Sequence[Union[Type[Detector], Detector, str]]]
        :param post_processor_list: The locale that the phone number should adhere to.
        :type post_processor_list: Optional[Sequence[Union[Type[Detector], Detector, str]]]
        :param locale: The locale of the documents in the format: 2 letter lower-case language code followed by an
                       underscore and the two letter upper-case country code, eg "en_GB" or "de_CH".
        :type locale: str, optional
        """
        super().__init__()

        # instantiate all of the detectors which, by default, uses all of the
        # detectors that are in the detectors.types dictionary
        self._detectors = {}  # type: Dict[str, Detector]
        self._post_processors = []  # type: List[PostProcessor]

        if locale is None:
            locale = 'en_US'
        self._locale = locale  # type: str

        if detector_list is None:
            # First we gather all detectors that should automatically load
            detector_list = [
                detector
                for detector in detectors.catalogue.detector_catalogue.get_all().values()
                if detector.autoload and (
                    # Then we filter out ones that don't support the current locale
                    not hasattr(detector, 'supported_locale') or (
                        hasattr(detector, 'supported_locale') and
                        detector.supported_locale(locale)  # type: ignore
                    )
                )
            ]

        for detector in detector_list:
            self.add_detector(detector, warn=True)

        if post_processor_list is None:
            post_processor_list = [
                post_processor
                for post_processor in sorted(
                    post_processors.catalogue.post_processor_catalogue.get_all().values(),
                    key=lambda pp: pp.index,
                )
                if post_processor.autoload
            ]

        for post_processor in post_processor_list:
            self.add_post_processor(post_processor)

```

--------------------------------

### Iterate Filth Documents with Scrubber

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub.html

Initialises a Scrubber with a specified locale and iterates over documents to find filth. Requires a list of documents and optional keyword arguments.

```python
scrubber = Scrubber(locale=locale)
return list(scrubber.iter_filth_documents(documents, **kwargs))
```

--------------------------------

### Find text between delimiters

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/tagged.html

Yields filth found between start and end substrings, supporting case-insensitivity, whitespace normalization, and word boundary constraints.

```python
) -> Generator[Filth, None, None]:
        """Yield filth for text between (and including)
        substr_start and substr_end, but only if the text
        between the two is less than limit characters.
        """
        text_orig = copy.copy(text)
        if ignore_case:
            text = text.lower()
            substr_start = substr_start.lower()
            substr_end = substr_end.lower()

        if ignore_whitespace:
            # We change any white space in the original with "\s+" that has to match one or more whitespace chars
            substr_start = '\\s+'.join([re.escape(token) for token in substr_start.split()])
            substr_end = '\\s+'.join([re.escape(token) for token in substr_end.split()])
        else:
            substr_start = re.escape(substr_start)
            substr_end = re.escape(substr_end)

        if ignore_partial_word_matches:
            substr_start = f"\\b{substr_start}\\b"
            substr_end = f"\\b{substr_end}\\b"

        matches = re.finditer(f"({substr_start})(.{{0,{limit}}})({substr_end})", text, re.MULTILINE | re.DOTALL)
        for match in matches:
            yield self.create_filth(
                match.span()[0],
                match.span()[1],
                text_orig[match.span()[0]:match.span()[1]],
                comparison_type=comparison_type,
                detector_name=self.name,
                document_name=document_name,
                locale=self.locale,
            )
```

--------------------------------

### Manage detectors by instance, class, or name

Source: https://scrubadub.readthedocs.io/en/stable/usage.html

Illustrates adding and removing detectors from a `Scrubber` instance using various methods: detector object, detector class, or detector name string. It also shows how to handle duplicate detector names.

```python
>>> import scrubadub
>>> # Create a detector with a name 'example_email'
>>> detector = scrubadub.detectors.EmailDetector(name='example_email')

>>> # Detectors can be added on Scrubber initialisation
>>> scrubber = scrubadub.Scrubber(detector_list=[detector, ])

>>> # add/remove a detector with a Detector instance
>>> scrubber.remove_detector(detector)

>>> # adds/removes detector with the default name 'email' using the class
>>> scrubber.add_detector(scrubadub.detectors.EmailDetector)
>>> scrubber.remove_detector(scrubadub.detectors.EmailDetector)

>>> # Adds/removes the scrubadub.detectors.EmailDetector detector, see the `name` attribute of the detector
>>> scrubber.add_detector('email')
>>> scrubber.remove_detector('email')

>>> # Add back the original instance
>>> scrubber.add_detector(detector)

>>> # KeyError is thrown if two detectors have the same name
>>> scrubber.add_detector(detector)
Traceback (most recent call last):
    ...
KeyError: 'can not add Detector "example_email" to this Scrubber, this name is already in use. Try removing it first.'
```

--------------------------------

### Define a Regex Detector

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/base.html

Example of creating a new detector using RegexDetector for pattern matching. Requires setting filth_cls and defining a regex attribute.

```python
>>> import re, scrubadub
>>> class NewUrlDetector(scrubadub.detectors.RegexDetector):
...     name = 'new_url_detector'
...     filth_cls = scrubadub.filth.url.UrlFilth
...     regex = re.compile(r'https.*$', re.IGNORECASE)
>>> scrubber = scrubadub.Scrubber(detector_list=[NewUrlDetector()])
>>> text = u"This url will be found https://example.com"
>>> scrubber.clean(text)
'This url will be found {{URL}}'
```

--------------------------------

### Initialize Scrubber with UserSuppliedFilthDetector

Source: https://scrubadub.readthedocs.io/en/stable/api_scrubadub_detectors.html

Use this detector to find known filth in text by providing a list of dictionaries. Each dictionary must contain 'match' and 'filth_type'. This detector is not enabled by default and must be added to the Scrubber.

```python
import scrubadub
scrubber = scrubadub.Scrubber(detector_list=[
    scrubadub.detectors.UserSuppliedFilthDetector([
        {'match': 'Anika', 'filth_type': 'name'},
        {'match': 'Larry', 'filth_type': 'name'},
    ]),
])
scrubber.clean("Anika is my favourite employee.")
```

--------------------------------

### PrefixSuffixReplacer Initialization

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/post_processors/prefix_suffix.html

Initializes the PrefixSuffixReplacer with optional prefix and suffix strings to be applied to filth replacements.

```APIDOC
## Class: PrefixSuffixReplacer

### Description
Initializes a post-processor that adds a prefix and/or suffix to the Filth's replacement string.

### Parameters
#### Constructor Parameters
- **prefix** (str) - Optional - The string to prepend to the replacement. Defaults to '{{'.
- **suffix** (str) - Optional - The string to append to the replacement. Defaults to '}}'.
- **name** (str) - Optional - A custom name for the post-processor.

### Usage Example
```python
from scrubadub.post_processors import PrefixSuffixReplacer
processor = PrefixSuffixReplacer(prefix='<b>', suffix='</b>')
```
```

--------------------------------

### Import necessary modules for scrubadub.comparison

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/comparison.html

Imports required modules for the scrubadub.comparison functionality, including regular expressions, copying, random data generation, itertools, Faker for synthetic data, and specific scrubadub modules like filth and detectors.

```python
import re
import copy
import random
import itertools

from faker import Faker

from . import filth as filth_module
from .filth import Filth
from .detectors.tagged import KnownFilthItem

from typing import List, Dict, Union, Optional, Tuple, Callable, Iterable, Type, Set
import numpy as np
import pandas as pd
import sklearn.metrics
```

--------------------------------

### Example Tagged PII Data Format

Source: https://scrubadub.readthedocs.io/en/stable/accuracy.html

This is the expected format for tagged PII data when using the TaggedEvaluationFilthDetector. Each item is a dictionary specifying the text to match and its filth type.

```json
[
    {"match": "wwashington@reed-ryan.org", "filth_type": "email"},
    {"match": "https://www.wong.com/", "filth_type": "url"}
]
```

--------------------------------

### Initialize Detector with Known Filth Items

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/tagged.html

Initializes a detector with a list of known filth items. Each item must be a dictionary containing 'match' and 'filth_type' keys. It validates input types and formats, and removes duplicate entries.

```python
super().__init__(**kwargs)

        for item in known_filth_items:
            if 'match' not in item or 'filth_type' not in item:
                raise KeyError("Each known filth item (dict) needs both keys 'match' and 'filth_type'.")
            if not isinstance(item['match'], str):
                raise ValueError("The value of 'match' in each KnownItem should be a string. "
                                 "Current value: " + item['match'].__repr__())
            if not isinstance(item['filth_type'], str):
                raise ValueError("The value of 'filth_type' in each KnownItem should be a string. "
                                 "Current value: " + item['filth_type'].__repr__())
            item['match'] = item['match'].strip()
            item['filth_type'] = item['filth_type'].strip()
            if 'match_end' in item:
                if not isinstance(item['match_end'], str):
                    raise ValueError("The value of 'match_end' in each KnownItem should be a string. "
                                     "Current value: " + item['match_end'].__repr__())
                item['match_end'] = item['match_end'].strip()

            for key in item.keys():
                if key not in ['match', 'match_end', 'limit', 'filth_type', 'ignore_case', 'ignore_whitespace',
                              'ignore_partial_word_matches']:
                    raise KeyError("Unexpected key '{}' in the known filth item.".format(key))

        self._known_filth_items = self.dedup_dicts(known_filth_items)
```

--------------------------------

### PrefixSuffixReplacer Initialization

Source: https://scrubadub.readthedocs.io/en/stable/api_scrubadub_post.html

Initializes the PrefixSuffixReplacer to wrap Filth replacement strings with specific prefixes and suffixes.

```APIDOC
## PrefixSuffixReplacer.__init__

### Description
Adds a prefix and/or suffix to the Filth’s replacement string.

### Parameters
#### Request Body
- **prefix** (str) - Optional - The string to prepend to the replacement.
- **suffix** (str) - Optional - The string to append to the replacement.
- **name** (str) - Optional - The name of the post-processor.
```

--------------------------------

### Create a Custom RegexDetector

Source: https://scrubadub.readthedocs.io/en/stable/api_scrubadub_detectors.html

Extend RegexDetector to create a new detector for specific patterns like URLs. Ensure to set the filth_cls and regex attributes. This example demonstrates detecting URLs with a case-insensitive regex.

```python
import re, scrubadub
>>> class NewUrlDetector(scrubadub.detectors.RegexDetector):
...     name = 'new_url_detector'
...     filth_cls = scrubadub.filth.url.UrlFilth
...     regex = re.compile(r'https.*$', re.IGNORECASE)
>>> scrubber = scrubadub.Scrubber(detector_list=[NewUrlDetector()])
>>> text = u"This url will be found https://example.com"
>>> scrubber.clean(text)
'This url will be found {{URL}}'
```

--------------------------------

### UserSuppliedFilthDetector Configuration Options

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/user_supplied.html

Illustrates different ways to configure the UserSuppliedFilthDetector with various options. These options control how the detector searches for and identifies filth in the text.

```python
>>> {'match': 'aaa', 'filth_type': 'name'}
```

```python
>>> {'match': 'aaa', 'match_end': 'zzz', 'filth_type': 'name'}
```

```python
>>> {'match': '012345', 'filth_type': 'phone', 'ignore_partial_word_matches': True}
```

--------------------------------

### Add an optional detector

Source: https://scrubadub.readthedocs.io/en/stable/usage.html

Demonstrates how to instantiate a Scrubber and register an optional detector to clean specific text patterns.

```python
>>> import scrubadub
>>> scrubber = scrubadub.Scrubber()
>>> scrubber.add_detector(scrubadub.detectors.DateOfBirthDetector)
>>> scrubber.clean("I was born on 5th December 1983")
'I was born {{DATE_OF_BIRTH}}'
```

--------------------------------

### Custom Filth Validation with RegexDetector

Source: https://scrubadub.readthedocs.io/en/stable/creating_detectors.html

Use Filth.is_valid() for complex validation beyond simple regex matching. This example defines a VisaCardNumberFilth class with Luhn algorithm validation and a VisaCardNumberDetector to find potential Visa card numbers.

```python
import scrubadub, string, stdnum.luhn, re

class VisaCardNumberFilth(scrubadub.filth.Filth):
    type = 'visa_card_number'

    def is_valid(self) -> bool:
        return stdnum.luhn.is_valid(self.text)

class VisaCardNumberDetector(scrubadub.detectors.RegexDetector):
    name = 'visa'
    filth_cls = VisaCardNumberFilth
    autoload = False
    regex = re.compile(r"\b4[0-9]{12}(?:[0-9]{3})?\b")

scrubber = scrubadub.Scrubber(detector_list=[VisaCardNumberDetector])
scrubber.clean('My reservation ID is 4343457563243982, please charge my room to the card 4829675671235307')
```

--------------------------------

### Get Filth Classification Report Function

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/comparison.html

This function generates a filth classification report from a list of Filth objects. It accepts parameters to control detector combination and document grouping. This is a high-level function for generating reports based on detected filth.

```python
[docs]
def get_filth_classification_report(
        filth_list: List[Filth],
        combine_detectors: bool = False,
        groupby_documents: bool = False,

```

--------------------------------

### Get Counts Method with Missing Expansion

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/comparison.html

This method calculates and returns the counts of filth items, optionally expanding the DataFrame to include all possible combinations of filth types and detectors. Use `expand_missing=True` to ensure a consistent DataFrame structure, which is useful for comparisons or further analysis.

```python
    def get_counts(self, expand_missing: bool = False) -> pd.DataFrame:
        if len(self.types) == 0:
            return pd.DataFrame()
        df_list = []  # type: List[pd.DataFrame]
        running_rows = 0
        for positioniser in self.types.values():
            pos_df = positioniser.get_counts()
            if expand_missing:
                pos_df = self.expand_missing(pos_df)
            df_list.append(pos_df)
            df_list[-1].index += running_rows
            running_rows += max(df_list[-1].index) + 1
        return pd.concat(df_list).fillna(0).astype(int)
```

--------------------------------

### Generate a filth classification report

Source: https://scrubadub.readthedocs.io/en/stable/accuracy.html

Demonstrates creating a fake document, running a detector, and generating a classification report to evaluate performance.

```python
>>> import scrubadub, scrubadub.comparison, scrubadub_spacy
>>> document, known_filth_items = scrubadub.comparison.make_fake_document(paragraphs=1, seed=3, filth_types=['name'])
>>> scrubber = scrubadub.Scrubber(detector_list=[scrubadub_spacy.detectors.SpacyNameDetector()])
>>> scrubber.add_detector(scrubadub.detectors.TaggedEvaluationFilthDetector(known_filth_items=known_filth_items))
>>> filth_list = list(scrubber.iter_filth(document))
>>> for filth in filth_list: print(filth)
<MergedFilth filths=[<NameFilth text='Jessica Sims' beg=112 end=124 detector_name='spacy_name' locale='en_US'>, <TaggedEvaluationFilth text='Jessica Sims' beg=112 end=124 comparison_type='name' detector_name='tagged' locale='en_US'>]>
<MergedFilth filths=[<NameFilth text='Michelle Mayer' beg=295 end=309 detector_name='spacy_name' locale='en_US'>, <TaggedEvaluationFilth text='Michelle Mayer' beg=295 end=309 comparison_type='name' detector_name='tagged' locale='en_US'>]>
<NameFilth text='nature activity' beg=363 end=378 detector_name='spacy_name' locale='en_US'>
<MergedFilth filths=[<NameFilth text='Claudia Carroll' beg=495 end=510 detector_name='spacy_name' locale='en_US'>, <TaggedEvaluationFilth text='Claudia Carroll' beg=495 end=510 comparison_type='name' detector_name='tagged' locale='en_US'>]>
<MergedFilth filths=[<NameFilth text='Laura Smith' beg=675 end=686 detector_name='spacy_name' locale='en_US'>, <TaggedEvaluationFilth text='Laura Smith' beg=675 end=686 comparison_type='name' detector_name='tagged' locale='en_US'>]>
>>> print(scrubadub.comparison.get_filth_classification_report(filth_list))
filth    detector      locale      precision    recall  f1-score   support

name     spacy_name    en_US            0.80      1.00      0.89         4

                        micro avg       0.80      1.00      0.89         4
                        macro avg       0.80      1.00      0.89         4
                     weighted avg       0.80      1.00      0.89         4
```

--------------------------------

### Configure Locale for Scrubadub

Source: https://scrubadub.readthedocs.io/en/stable/localization.html

Demonstrates setting the locale parameter in clean functions and the Scrubber class to handle region-specific phone number formats.

```python
>>> import scrubadub
>>> scrubadub.clean('My US number is 731-938-1630', locale='en_US')
'My US number is {{PHONE}}'
>>> scrubadub.clean('My US number is 731-938-1630', locale='en_GB')
'My US number is 731-938-1630'
>>> scrubadub.clean('My GB number is 0121 496 0112', locale='en_GB')
'My GB number is {{PHONE}}'
>>> scrubadub.clean('My GB number is 0121 496 0112', locale='en_US')
'My GB number is 0121 496 0112'
>>> scrubber = scrubadub.Scrubber(locale='de_DE')
>>> scrubber.clean('Meine Telefonnummer ist 05086 63680')
'Meine Telefonnummer ist {{PHONE}}'
```

--------------------------------

### Initialize Detector with Custom Name

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/base.html

Demonstrates how the Detector class is initialized, including handling custom names and locales. A warning is issued if the detector name is not explicitly set and derived from filth_cls.type.

```python
if getattr(self, 'name', 'detector') == 'detector' and getattr(self, 'filth_cls', None) is not None:
            if getattr(self.filth_cls, 'type', None) is not None and type(self) != Detector:
                self.name = self.filth_cls.type
                warnings.warn(
                    "Setting the detector name from the filth_cls.type is depreciated, please declare an explicit name"
                    "attribute on the class.",
                    DeprecationWarning
                )
        if name is not None:
            self.name = name

        self.locale = locale
        self.language, self.region = self.locale_split(locale)

        if hasattr(self, 'supported_locale'):
            if not self.supported_locale(locale=locale):  # type: ignore
                warnings.warn("Detector {} does not support the locale '{}'வுகளை.".format(self.name, locale))
```

--------------------------------

### Add an external detector

Source: https://scrubadub.readthedocs.io/en/stable/usage.html

Shows the process of importing an external detector package and adding it to the Scrubber instance.

```python
>>> import scrubadub, scrubadub_spacy
>>> scrubber = scrubadub.Scrubber()
>>> scrubber.add_detector(scrubadub_spacy.detectors.SpacyEntityDetector)
>>> scrubber.clean("My name is John")
'My name is {{NAME}}'
```

--------------------------------

### Add Detectors to Scrubber

Source: https://scrubadub.readthedocs.io/en/stable/api_scrubadub.html

Demonstrates adding detectors to a Scrubber instance using class types, detector names, or pre-initialized detector instances.

```python
>>> import scrubadub
>>> scrubber = scrubadub.Scrubber(detector_list=[])
>>> scrubber.add_detector(scrubadub.detectors.CreditCardDetector)
>>> scrubber.add_detector('skype')
>>> detector = scrubadub.detectors.DateOfBirthDetector(require_context=False)
>>> scrubber.add_detector(detector)
```

--------------------------------

### Detector Class Initialization

Source: https://scrubadub.readthedocs.io/en/stable/_modules/scrubadub/detectors/base.html

Initializes a new detector instance with a specific name and locale.

```APIDOC
## __init__

### Description
Initializes the Detector instance with an optional name and locale.

### Parameters
#### Path Parameters
- **name** (str) - Optional - Overrides the default name of the Detector.
- **locale** (str) - Optional - The locale of the documents (e.g., 'en_US', 'en_GB').

### Request Example
{
  "name": "my_custom_detector",
  "locale": "en_US"
}
```
Tokens:
Raw