### Example Usage of TokenMatcher Initialization and Rule Addition Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Demonstrates the basic setup of a TokenMatcher by initializing it with a spaCy vocabulary, adding a rule with a fuzzy text match, and then checking for the rule's presence. ```python import spacy from spaczz.matcher import TokenMatcher nlp = spacy.blank("en") matcher = TokenMatcher(nlp.vocab) matcher.add("AUTHOR", [[{"TEXT": {"FUZZY": "Kerouac"}}]]) "AUTHOR" in matcher ``` -------------------------------- ### Similarity Matcher Example Source: https://github.com/gandersen101/spaczz/blob/main/notebooks/README.ipynb Demonstrates lowering min_r2 to produce matches with the SimilarityMatcher. Requires spaCy and spaczz to be installed. ```python import spacy from spaczz.matcher import SimilarityMatcher nlp = spacy.load("en_core_web_sm") # lowering min_r2 from default of 75 to produce matches in this example matcher = SimilarityMatcher(nlp.vocab, min_r2=65) matcher.add("FRUIT", [nlp("fruit")]) doc = nlp("apples, grapes, and bananas are fruits.") matches = matcher(doc) for match_id, start, end, ratio, pattern in matches: print(match_id, doc[start:end], ratio, pattern) ``` -------------------------------- ### Install spaczz and Dev Dependencies with Poetry Source: https://github.com/gandersen101/spaczz/blob/main/notebooks/README.ipynb Install the project and its development dependencies using Poetry. This command should be run from the root directory of the spaczz project. ```bash poetry install # Within spaczz's root directory. ``` -------------------------------- ### Install Spaczz Development Dependencies Source: https://github.com/gandersen101/spaczz/blob/main/README.md Install Spaczz and its development dependencies using Poetry. This command should be run within the Spaczz root directory. ```bash poetry install ``` -------------------------------- ### Install spaczz Source: https://github.com/gandersen101/spaczz/blob/main/docs/index.md Install spaczz using pip. This command is used to add the library to your Python environment. ```Python pip install spaczz ``` -------------------------------- ### Initialize RegexMatcher Source: https://context7.com/gandersen101/spaczz/llms.txt Sets up a spaCy pipeline with Spaczz's RegexMatcher. This snippet is a starting point for using predefined regex patterns. ```python import spacy from spaczz.matcher import RegexMatcher nlp = spacy.blank("en") ``` -------------------------------- ### Initialize and Use FuzzyMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Demonstrates how to initialize FuzzyMatcher, add a pattern, and find matches within a spaCy Doc object. Ensure you have spaCy and spaczz installed. ```python import spacy from spaczz.matcher import FuzzyMatcher nlp = spacy.blank("en") matcher = FuzzyMatcher(nlp.vocab) doc = nlp("Rdley Scott was the director of Alien.") matcher.add("NAME", [nlp.make_doc("Ridley Scott")]) matcher(doc) [('NAME', 0, 2, 96, 'Ridley Scott')] ``` -------------------------------- ### Example Usage of TokenMatcher Labels Property Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Illustrates how to initialize a TokenMatcher, add a rule, and then access the 'labels' property to retrieve a tuple of all added rule labels. ```python import spacy from spaczz.matcher import TokenMatcher nlp = spacy.blank("en") matcher = TokenMatcher(nlp.vocab) matcher.add("AUTHOR", [[{"TEXT": {"FUZZY": "Kerouac"}}]]) matcher.labels ``` -------------------------------- ### Install Git Pre-commit Hooks Source: https://github.com/gandersen101/spaczz/blob/main/README.md Install Git pre-commit hooks to ensure code quality and consistency before committing changes. This is typically run after installing development dependencies. ```bash pre-commit install ``` -------------------------------- ### Initialize and Use TokenMatcher with spaCy Source: https://github.com/gandersen101/spaczz/blob/main/README.md Initialize TokenMatcher with a spaCy vocabulary and add patterns for matching. This example demonstrates matching sequences of tokens with fuzzy and fuzzy-regex criteria. ```python import spacy from spaczz.matcher import TokenMatcher # Using model results like POS tagging in token patterns requires model that provides these. nlp = spacy.load("en_core_web_md") text = """The manager gave me SQL databesE acess so now I can acces the Sequal DB. My manager's name is Grfield""" doc = nlp(text) matcher = TokenMatcher(vocab=nlp.vocab) matcher.add( "DATA", [ [ {"TEXT": "SQL"}, {"LOWER": {"FREGEX": "(database){s<=1}"}}, {"LOWER": {"FUZZY": "access"}}, ], [{"TEXT": {"FUZZY": "Sequel"}, "POS": "PROPN"}, {"LOWER": "db"}], ], ) matcher.add("NAME", [[{"TEXT": {"FUZZY": "Garfield"}}]]) matches = matcher(doc) for match_id, start, end, ratio, pattern in matches: print(match_id, doc[start:end], ratio, pattern) ``` -------------------------------- ### SpaczzRuler Token Pattern Example Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Example of a token pattern configuration for the SpaczzRuler, using a list of dictionaries to define token attributes and fuzzy matching. ```default { 'label': 'ORG', 'pattern': [{'TEXT': {'FUZZY': 'Apple'}}], 'type': 'token', } ``` -------------------------------- ### Add Fuzzy Pattern to SpaczzRuler Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Demonstrates adding a fuzzy pattern for 'AUTHOR' to the SpaczzRuler. Ensure spaCy and Spaczz are installed. ```python >>> import spacy >>> from spaczz.pipeline import SpaczzRuler >>> nlp = spacy.blank("en") >>> ruler = SpaczzRuler(nlp) >>> ruler.add_patterns([{"label": "AUTHOR", "pattern": "Kerouac", "type": "fuzzy"}]) >>> ruler.labels ('AUTHOR',) ``` -------------------------------- ### Example Usage of TokenMatcher Patterns Property Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Shows the initialization of a TokenMatcher, adding a rule, and then accessing the 'patterns' property to inspect the stored rule configurations, including label, pattern, and type. ```python import spacy from spaczz.matcher import TokenMatcher nlp = spacy.blank("en") matcher = TokenMatcher(nlp.vocab) matcher.add("AUTHOR", [[{"TEXT": {"FUZZY": "Kerouac"}}]]) matcher.patterns == [ { "label": "AUTHOR", "pattern": [{"TEXT": {"FUZZY": "Kerouac"}}], "type": "token", }, ] ``` -------------------------------- ### Using Predefined Regex Patterns Source: https://github.com/gandersen101/spaczz/blob/main/README.md Shows how to add predefined regex patterns to the RegexMatcher using their key names. This example uses the 'street_addresses' predefined pattern. ```python import spacy from spaczz.matcher import RegexMatcher nlp = spacy.blank("en") text = """Anderson, Grint created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the USA.""" # Spelling errors intentional. Notice 'USA' here. doc = nlp(text) matcher = RegexMatcher(nlp.vocab) # Use inline flags for regex strings as needed matcher.add( "STREET", ["street_addresses"], kwargs=[{"predef": True}] ) # Use predefined regex by key name. ``` -------------------------------- ### Using Predefined Patterns with RegexMatcher Source: https://context7.com/gandersen101/spaczz/llms.txt Demonstrates how to use predefined patterns like 'emails', 'phones', and 'dates' with RegexMatcher to extract entities from text. Ensure the 'spaczz' library is installed. ```python import spacy from spaczz.matcher import RegexMatcher predefined_patterns = [ "dates", # Date patterns (Jan 1, 2024, 01/01/2024, etc.) "times", # Time patterns (12:30, 3pm, etc.) "phones", # Phone numbers "phones_with_exts", # Phone numbers with extensions "links", # URLs and web links "emails", # Email addresses "ips", # IPv4 addresses "ipv6s", # IPv6 addresses "prices", # Price values ($99.99) "hex_colors", # Hex color codes (#FF0000) "credit_cards", # Credit card numbers "btc_addresses", # Bitcoin addresses "street_addresses", # Street addresses "zip_codes", # ZIP codes (US format) "po_boxes", # PO Box addresses "ssn_numbers", # Social Security Numbers ] text = """Contact: john@example.com, 555-123-4567 Address: 123 Main Street, 90210 Price: $299.99 Date: January 15, 2024 IP: 192.168.1.1""" doc = nlp(text) matcher = RegexMatcher(nlp.vocab) matcher.add("EMAIL", ["emails"], kwargs=[{"predef": True}]) matcher.add("PHONE", ["phones"], kwargs=[{"predef": True}]) matcher.add("STREET", ["street_addresses"], kwargs=[{"predef": True}]) matcher.add("ZIP", ["zip_codes"], kwargs=[{"predef": True}]) matcher.add("PRICE", ["prices"], kwargs=[{"predef": True}]) matcher.add("DATE", ["dates"], kwargs=[{"predef": True}]) matcher.add("IP", ["ips"], kwargs=[{"predef": True}]) matches = matcher(doc) for match_id, start, end, ratio, pattern in sorted(matches, key=lambda x: x[1]): print(f"{match_id}: '{doc[start:end]}'") # Output: # EMAIL: 'john@example.com' # PHONE: '555-123-4567' # STREET: '123 Main Street' # ZIP: '90210' # PRICE: '$299.99' # DATE: 'January 15, 2024' # IP: '192.168.1.1' ``` -------------------------------- ### Basic Fuzzy Matching with FuzzyMatcher Source: https://context7.com/gandersen101/spaczz/llms.txt Demonstrates basic fuzzy phrase matching using FuzzyMatcher. Ideal for finding misspelled names or entities with character-level variations. Requires spaCy and Spaczz installation. ```python import spacy from spacy.tokens import Span from spaczz.matcher import FuzzyMatcher nlp = spacy.blank("en") # Text with intentional spelling errors text = """Grint M Anderson created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the US.""" doc = nlp(text) # Basic fuzzy matching matcher = FuzzyMatcher(nlp.vocab) matcher.add("NAME", [nlp("Grant Andersen")]) matcher.add("GPE", [nlp("Nashville")]) matches = matcher(doc) for match_id, start, end, ratio, pattern in matches: print(f"Label: {match_id}, Text: {doc[start:end]}, Ratio: {ratio}, Pattern: {pattern}") ``` -------------------------------- ### Get All Patterns from TokenMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Access all patterns and their associated match settings that have been added to the TokenMatcher. The output is a list of dictionaries, each detailing a rule. ```python matcher.patterns == [ { "label": "AUTHOR", "pattern": [{"TEXT": {"FUZZY": "Kerouac"}}], "type": "token", }, ] ``` -------------------------------- ### RegexMatcher pattern matching Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Shows how to use RegexMatcher to find patterns within a spaCy Doc object. This example demonstrates adding a regex pattern and retrieving match results, including label, start/end indices, match ratio, and the pattern itself. ```python >>> import spacy >>> from spaczz.matcher import RegexMatcher >>> nlp = spacy.blank("en") >>> matcher = RegexMatcher(nlp.vocab) >>> doc = nlp.make_doc("I live in the united states, or the US") >>> matcher.add("GPE", ["[Uu](nited|\.?) ?[Ss](tates|\.?)?"]) >>> matcher(doc)[0] ('GPE', 4, 6, 100, '[Uu](nited|\.?) ?[Ss](tates|\.?)?') ``` -------------------------------- ### Download spaCy English Model Source: https://github.com/gandersen101/spaczz/blob/main/README.md Download the spaCy medium English model ('en_core_web_md') required for testing and documentation examples. This command is run using Poetry's execution context. ```bash poetry run python -m spacy download "en_core_web_md" ``` -------------------------------- ### Save SpaczzRuler Patterns to Disk Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Saves the SpaczzRuler patterns to a specified directory as newline-delimited JSON (JSONL). Requires temporary directory setup for testing. ```python >>> import os >>> import tempfile >>> import spacy >>> from spaczz.pipeline import SpaczzRuler >>> nlp = spacy.blank("en") >>> ruler = SpaczzRuler(nlp) >>> ruler.add_patterns([{"label": "AUTHOR", "pattern": "Kerouac", "type": "fuzzy"}]) >>> with tempfile.TemporaryDirectory() as tmpdir: >>> ruler.to_disk(f"{tmpdir}/ruler") >>> isdir = os.path.isdir(f"{tmpdir}/ruler") >>> isdir True ``` -------------------------------- ### Add Pattern to SimilarityMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Shows how to add a simple pattern to the SimilarityMatcher without any specific keyword arguments. This example also verifies if the added label exists in the matcher. ```python import spacy from spaczz.matcher import SimilarityMatcher nlp = spacy.load("en_core_web_md") matcher = SimilarityMatcher(nlp.vocab) matcher.add("SOUND", [nlp("mooo")]) "SOUND" in matcher ``` -------------------------------- ### Example Usage of TokenMatcher Remove Method Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Demonstrates the process of adding a rule to a TokenMatcher, subsequently removing it by its label, and then verifying its removal by checking for the label's existence. ```python import spacy from spaczz.matcher import TokenMatcher nlp = spacy.blank("en") matcher = TokenMatcher(nlp.vocab) matcher.add("AUTHOR", [[{"TEXT": {"FUZZY": "Kerouac"}}]]) matcher.remove("AUTHOR") "AUTHOR" in matcher ``` -------------------------------- ### Programmatically Generate Fuzzy Patterns with Min Ratio Source: https://github.com/gandersen101/spaczz/blob/main/notebooks/fuzzy_matching_tweaks.ipynb Generate fuzzy matching patterns from raw data, applying different minimum match ratios ('min_r') based on pattern length. Shorter patterns get a higher 'min_r' to enforce stricter matching. ```python raw_patterns = srsly.read_json(path / "countries.json") fuzzy_patterns = [] for pattern in raw_patterns: template = { "label": "COUNTRY", "pattern": pattern["name"], "type": "fuzzy", "id": pattern["name"], } if len(template["pattern"]) < 5: template["kwargs"] = {"min_r": 100} # see note above elif len(template["pattern"]) >= 5 and len(template["pattern"]) < 8: template["kwargs"] = {"min_r": 85} fuzzy_patterns.append(template) ``` -------------------------------- ### Find fuzzy token matches with TokenMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Initialize a TokenMatcher with the vocabulary and add labeled patterns. Then, call the matcher on a spaCy Doc object to retrieve matches, which are returned as tuples containing label, start index, end index, and match ratio. ```python import spacy >>> from spaczz.matcher import TokenMatcher >>> nlp = spacy.blank("en") >>> matcher = TokenMatcher(nlp.vocab) >>> doc = nlp("Rdley Scot was the director of Alien.") >>> matcher.add("NAME", [ [{"TEXT": {"FUZZY": "Ridley"}}, {"TEXT": {"FUZZY": "Scott"}}] ]) >>> matcher(doc)[0][:4] ('NAME', 0, 2, 90) ``` -------------------------------- ### SpaczzRuler Initialization and Basic Usage Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Demonstrates how to initialize SpaczzRuler and add patterns for fuzzy matching. ```APIDOC ## SpaczzRuler Initialization and Basic Usage ### Description This section covers the initialization of the `SpaczzRuler` and how to add patterns for fuzzy matching. ### Method `__init__` ### Endpoint N/A (Class constructor) ### Parameters - **nlp** (Language) - The spaCy language object. - **patterns** (Sequence[Dict[str, str | Dict[str, Any] | List[Dict[str, Any]]], optional) - Initial list of patterns to add. ### Request Example ```python import spacy from spaczz.pipeline import SpaczzRuler nlp = spacy.blank("en") ruler = SpaczzRuler(nlp) doc = nlp.make_doc("My name is Grant Andersen") ruler.add_patterns([{"label": "NAME", "pattern": "Grant Andersen", "type": "fuzzy"}]) doc = ruler(doc) print([ent.text for ent in doc.ents]) ``` ### Response #### Success Response (200) N/A (Constructor does not return a value directly, but initializes the object). #### Response Example N/A ``` -------------------------------- ### SpaczzRuler Fuzzy Phrase Pattern Example Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Example of a fuzzy phrase pattern configuration for the SpaczzRuler, specifying label, pattern, type, and optional keyword arguments. ```default { 'label': 'ORG', 'pattern': 'Apple', 'kwargs': {'min_r2': 90}, 'type': 'fuzzy', } ``` -------------------------------- ### Get Labels from RegexMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Retrieves all unique labels currently stored in the RegexMatcher as a tuple of strings. ```python >>> matcher.labels ('ZIP',) ``` -------------------------------- ### Initialize and Use SpaczzRuler Source: https://github.com/gandersen101/spaczz/blob/main/README.md Demonstrates initializing SpaczzRuler with a spaCy model, adding various pattern types (fuzzy, regex, token), and processing a document to identify entities with custom attributes. ```python import spacy from spaczz.pipeline import SpaczzRuler nlp = spacy.blank("en") text = """Anderson, Grint created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the USA. Some of his favorite bands are Converg and Protet the Zero.""" # Spelling errors intentional. doc = nlp(text) patterns = [ { "label": "NAME", "pattern": "Grant Andersen", "type": "fuzzy", "kwargs": {"fuzzy_func": "token_sort"}, }, { "label": "STREET", "pattern": "street_addresses", "type": "regex", "kwargs": {"predef": True}, }, {"label": "GPE", "pattern": "Nashville", "type": "fuzzy"}, { "label": "ZIP", "pattern": r"\b(?:55554){s<=1}(?:(?:[-\s])?\d{4}\b)", "type": "regex", }, # fuzzy regex {"label": "GPE", "pattern": "(?i)[U](nited|\.?) ?[S](tates|\.?) ", "type": "regex"}, { "label": "BAND", "pattern": [{"LOWER": {"FREGEX": "(converge){e<=1}" }}], "type": "token", }, { "label": "BAND", "pattern": [ {"TEXT": {"FUZZY": "Protest"}}, {"IS_STOP": True}, {"TEXT": {"FUZZY": "Hero"}}, ], "type": "token", }, ] ruler = SpaczzRuler(nlp) ruler.add_patterns(patterns) doc = ruler(doc) for ent in doc.ents: print( ( ent.text, ent.start, ent.end, ent.label_, ent._.spaczz_ratio, ent._.spaczz_type, ent._.spaczz_pattern, ) ) ``` -------------------------------- ### SimilarityMatcher Initialization and Defaults Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Explains the default match settings for the SimilarityMatcher and how they can be overridden. ```APIDOC ## SimilarityMatcher Defaults ### Description Keyword arguments to be used as default match settings. Per-pattern match settings take precedence over defaults. ### Type dict[str, bool|int|str|Literal['default', 'min', 'max']] ### Match Settings - **ignore_case** (bool) - Whether to lower-case text before fuzzy matching. Default is True. - **min_r** (int) - Minimum match ratio required. - **thresh** (int) - If this ratio is exceeded in initial scan, and flex > 0, no optimization will be attempted. If flex == 0, thresh has no effect. Default is 100. - **flex** (int|Literal['default', 'min', 'max']) - Number of tokens to move match boundaries left and right during optimization. Can be an int with a max of len(pattern) and a min of 0, (will warn and change if higher or lower). "max", "min", or "default" are also valid. Default is "default": len(pattern) // 2. - **min_r1** (int|None) - Optional granular control over the minimum match ratio required for selection during the initial scan. If flex == 0, min_r1 will be overwritten by min_r2. If flex > 0, min_r1 must be lower than min_r2 and "low" in general because match boundaries are not flexed initially. Default is None, which will result in min_r1 being set to round(min_r / 1.5). - **min_r2** (int|None) - Optional granular control over the minimum match ratio required for selection during match optimization. Needs to be higher than min_r1 and "high" in general to ensure only quality matches are returned. Default is None, which will result in min_r2 being set to min_r. ``` -------------------------------- ### Get labels from FuzzyMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Shows how to retrieve all unique labels currently stored in the FuzzyMatcher. This is helpful for inspecting the state of the matcher. ```python >>> import spacy >>> from spaczz.matcher import FuzzyMatcher >>> nlp = spacy.blank("en") >>> matcher = FuzzyMatcher(nlp.vocab) >>> matcher.add("AUTHOR", [nlp.make_doc("Kerouac")]) >>> matcher.labels ('AUTHOR',) ``` -------------------------------- ### Initialize and Use SimilarityMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Demonstrates how to initialize the SimilarityMatcher with a spaCy vocabulary, add a pattern with custom match settings, and then use the matcher to find matches in a document. Ensure you have loaded a spaCy model with vectors (e.g., 'en_core_web_md'). ```python import spacy from spaczz.matcher import SimilarityMatcher nlp = spacy.load("en_core_web_md") matcher = SimilarityMatcher(nlp.vocab) doc = nlp("I like apples.") matcher.add("FRUIT", [nlp("fruit")], [{"min_r": 60}]) matcher(doc) ``` -------------------------------- ### Get All Labels from TokenMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Retrieve all unique labels currently stored in the TokenMatcher as a tuple of strings. This can be used to inspect the rules that have been added. ```python matcher.labels ``` -------------------------------- ### Load SpaczzRuler from Disk Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Shows how to save a SpaczzRuler to disk using `to_disk` and then load it back into a new ruler instance using `from_disk`. ```python >>> import os >>> import tempfile >>> import spacy >>> from spaczz.pipeline import SpaczzRuler >>> nlp = spacy.blank("en") >>> ruler = SpaczzRuler(nlp) >>> ruler.add_patterns([{"label": "AUTHOR", "pattern": "Kerouac", "type": "fuzzy"}]) >>> with tempfile.TemporaryDirectory() as tmpdir: >>> ruler.to_disk(f"{tmpdir}/ruler") >>> new_ruler = SpaczzRuler(nlp) >>> new_ruler = new_ruler.from_disk(f"{tmpdir}/ruler") >>> "AUTHOR" in new_ruler True ``` -------------------------------- ### SpaczzRuler Training Initialization Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Details the `initialize` method used for setting up the ruler for training. ```APIDOC ## SpaczzRuler Training Initialization ### Description This section describes the `initialize` method, which is used to set up the `SpaczzRuler` component for training purposes. ### Method `initialize(get_examples: Callable[[], Iterable[Example]], nlp: Language | None = None, patterns: Sequence[Dict[str, str | Dict[str, Any] | List[Dict[str, Any]]]] | None = None) → None` ### Parameters - **get_examples** (Callable[[], Iterable[Example]]) - Function that returns a representative sample of gold-standard Example objects. - **nlp** (Language, optional) - The current nlp object the component is part of. - **patterns** (Sequence[Dict[str, str | Dict[str, Any] | List[Dict[str, Any]]]], optional) - The list of patterns. ``` -------------------------------- ### Get Patterns from RegexMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Retrieves all patterns and their associated match settings from the RegexMatcher. The output is a list of dictionaries, each detailing the label, pattern, type, and kwargs. ```python >>> matcher.patterns == [ { "label": "ZIP", "pattern": "zip_codes", "type": "regex", "kwargs": {"predef": True}, } ] True ``` -------------------------------- ### Iterate FuzzyMatcher Results Source: https://github.com/gandersen101/spaczz/blob/main/README.md Iterate through the matches found by the FuzzyMatcher to access match ID, start and end indices, similarity ratio, and the matched pattern. ```python for match_id, start, end, ratio, pattern in matches: print( match_id, doc[start:end], ratio, pattern ) # comma in result isn't ideal - see "Roadmap" ``` -------------------------------- ### Initialize spaCy and add SpaczzRuler Source: https://github.com/gandersen101/spaczz/blob/main/notebooks/fuzzy_matching_tweaks.ipynb Initializes a blank English spaCy pipeline and adds the 'spaczz_ruler' component, then adds the pre-formatted fuzzy patterns. ```python nlp = spacy.blank("en") spaczz_ruler = nlp.add_pipe("spaczz_ruler") spaczz_ruler.add_patterns(fuzzy_patterns) ``` -------------------------------- ### Fuzzy Matching with Match Settings Source: https://context7.com/gandersen101/spaczz/llms.txt Illustrates using FuzzyMatcher with various settings like 'ignore_case', 'min_r', and 'flex' to control matching strictness and boundary flexibility. This is useful for handling variations in entity spelling. ```python import spacy from spaczz.matcher import FuzzyMatcher nlp = spacy.blank("en") text = "The Untied States of America" doc = nlp(text) # Key parameters explained: # - ignore_case (bool): Lowercase before matching. Default: True # - min_r (int): Minimum match ratio required (0-100) # - min_r1 (int): Minimum ratio for initial scan (default: min_r / 1.5) # - min_r2 (int): Minimum ratio for optimization (default: min_r) # - thresh (int): Skip optimization if exceeded. Default: 100 # - flex (int|'default'|'min'|'max'): Token boundary flexibility # Example: Strict matching with high minimum ratio strict_matcher = FuzzyMatcher(nlp.vocab) strict_matcher.add("GPE", [nlp("United States")], kwargs=[ { "min_r": 90, # Require 90% match "flex": 0, # No boundary adjustment }, ]) # Example: Lenient matching with lower threshold lenient_matcher = FuzzyMatcher(nlp.vocab) lenient_matcher.add("GPE", [nlp("United States")], kwargs=[ { "min_r": 70, # Accept 70% match "flex": "max", # Maximum boundary flexibility "min_r1": 50, # Low initial scan threshold }, ]) strict_matches = strict_matcher(doc) lenient_matches = lenient_matcher(doc) print(f"Strict matches: {len(strict_matches)}") print(f"Lenient matches: {len(lenient_matches)}") if lenient_matches: _, start, end, ratio, _ = lenient_matches[0] print(f" Found: '{doc[start:end]}' with ratio {ratio}") ``` -------------------------------- ### Using Predefined Patterns with RegexMatcher Source: https://context7.com/gandersen101/spaczz/llms.txt Demonstrates the use of predefined regex patterns for common entities like emails, street addresses, and prices with RegexMatcher. Set `predef=True` to enable. ```python matcher2 = RegexMatcher(nlp.vocab) matcher2.add("EMAIL", ["emails"], kwargs=[{"predef": True}]) matcher2.add("STREET", ["street_addresses"], kwargs=[{"predef": True}]) matcher2.add("PRICE", ["prices"], kwargs=[{"predef": True}]) matches2 = matcher2(doc) for match_id, start, end, ratio, pattern in matches2: print(f"Predefined {match_id}: '{doc[start:end]}'") ``` -------------------------------- ### Basic RegexMatcher Usage with Fuzzy Matching Source: https://github.com/gandersen101/spaczz/blob/main/README.md Demonstrates adding regex patterns to the RegexMatcher and performing fuzzy matching. Ensure variables for ratio and pattern are included when unpacking results. ```python import spacy from spaczz.matcher import RegexMatcher nlp = spacy.blank("en") text = """Anderson, Grint created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the US.""" # Spelling errors intentional. doc = nlp(text) matcher = RegexMatcher(nlp.vocab) # Use inline flags for regex strings as needed matcher.add( "ZIP", [r"\b\d{5}(?:[-\s]\d{4})?\b"], ) matcher.add("GPE", [r"(usa){d<=1}"]) # Fuzzy regex. matches = matcher(doc) for match_id, start, end, ratio, pattern in matches: print(match_id, doc[start:end], ratio, pattern) ``` -------------------------------- ### Test default fuzzy matching Source: https://github.com/gandersen101/spaczz/blob/main/notebooks/fuzzy_matching_tweaks.ipynb Processes a test string with the default SpaczzRuler configuration and prints the found countries. This example highlights potential issues with multi-word matches. ```python doc = nlp("This is a test that should find Egypt and Argentina") countries = [(ent.ent_id_, ent.text) for ent in doc.ents if ent.label_ == "COUNTRY"] if countries != [("Egypt", "Egypt"), ("Argentina", "Argentina")]: print("Unexpected results...") print(countries) else: print("Success!") print(countries) ``` -------------------------------- ### Access patterns in FuzzyMatcher Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Illustrates how to access all patterns and their associated match settings that have been added to the FuzzyMatcher. This is useful for debugging or understanding the matcher's configuration. ```python >>> import spacy >>> from spaczz.matcher import FuzzyMatcher >>> nlp = spacy.blank("en") >>> matcher = FuzzyMatcher(nlp.vocab) >>> matcher.add("AUTHOR", [nlp.make_doc("Kerouac")], [{"ignore_case": False}]) >>> matcher.patterns == [ { "label": "AUTHOR", "pattern": "Kerouac", "type": "fuzzy", "kwargs": {"ignore_case": False} }, ] True ``` -------------------------------- ### Compare Fuzzy Matching Functions Source: https://context7.com/gandersen101/spaczz/llms.txt Illustrates how different fuzzy matching algorithms in Spaczz (e.g., 'simple', 'token_sort', 'token_set') affect matching results for reordered names. Requires RapidFuzz library. ```python import spacy from spaczz.matcher import FuzzyMatcher nlp = spacy.blank("en") # Available fuzzy functions and their use cases: fuzzy_functions = { "simple": "Basic character-level ratio (default)", "partial": "Best partial string match", "token": "Token-based ratio", "token_set": "Compares token sets, ignoring order/duplicates", "token_sort": "Sorts tokens alphabetically before comparing", "partial_token": "Partial match on tokens", "partial_token_set": "Partial match on token sets", "partial_token_sort": "Partial match with sorted tokens", "weighted": "Weighted ratio (WRatio)", "quick": "Quick ratio (QRatio)", "partial_alignment": "Partial ratio with alignment (rapidfuzz>=2.0.3)", } # Example: Comparing different fuzzy functions text = "Anderson Grant is here" doc = nlp(text) pattern = nlp("Grant Anderson") for func_name in ["simple", "token_sort", "token_set", "partial"]: matcher = FuzzyMatcher(nlp.vocab) matcher.add("TEST", [pattern], kwargs=[{"fuzzy_func": func_name}]) matches = matcher(doc) if matches: _, start, end, ratio, _ = matches[0] print(f"{func_name}: '{doc[start:end]}' - Ratio: {ratio}") else: print(f"{func_name}: No match") # Output shows how token_sort handles reordered names: # simple: No match (or low ratio) # token_sort: 'Anderson Grant' - Ratio: 100 # token_set: 'Anderson Grant' - Ratio: 100 # partial: 'Grant' - Ratio: 100 ``` -------------------------------- ### TokenMatcher with MIN_R Setting Source: https://context7.com/gandersen101/spaczz/llms.txt Configures TokenMatcher with a minimum ratio (MIN_R=85) and partial fuzzy matching for 'access' in a regex pattern for 'database'. ```python matcher2 = TokenMatcher(vocab=nlp.vocab) matcher2.add( "TECH", [ [ {"TEXT": {"FREGEX": "(database){e<=1}"}}, {"LOWER": {"FUZZY": "access", "MIN_R": 85, "FUZZY_FUNC": "partial"}}, ] ], ) # Check matcher properties print(f"Labels: {matcher.labels}") # ('DATA', 'NAME') print(f"Patterns: {matcher.patterns}") ``` -------------------------------- ### Fuzzy Regex Matching with Edit Distance Source: https://context7.com/gandersen101/spaczz/llms.txt Illustrates fuzzy regex matching using the `regex` library's syntax, allowing for approximate matches with a specified edit distance. Example uses `{{d<=1}}` for up to 1 deletion. ```python text2 = "Visit the USA or the US today." doc2 = nlp(text2) matcher3 = RegexMatcher(nlp.vocab) matcher3.add("GPE", [r"(usa){{d<=1}}"]) # Fuzzy regex allows 1 deletion matches3 = matcher3(doc2) for match_id, start, end, ratio, pattern in matches3: print(f"Fuzzy regex: '{doc2[start:end]}', Ratio: {ratio}") ``` -------------------------------- ### FuzzyMatcher with On-Match Callback Source: https://github.com/gandersen101/spaczz/blob/main/README.md Shows how to use on-match callback functions with FuzzyMatcher to add custom entities to the document. The callback receives the matcher, doc, match index, and matches. ```python import spacy from spacy.tokens import Span from spaczz.matcher import FuzzyMatcher nlp = spacy.blank("en") text = """Grint M Anderson created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the US.""" # Spelling errors intentional. doc = nlp(text) def add_name_ent(matcher, doc, i, matches): """Callback on match function. Adds "NAME" entities to doc.""" # Get the current match and create tuple of entity label, start and end. # Append entity to the doc's entity. (Don't overwrite doc.ents!) _match_id, start, end, _ratio, _pattern = matches[i] entity = Span(doc, start, end, label="NAME") doc.ents += (entity,) matcher = FuzzyMatcher(nlp.vocab) matcher.add("NAME", [nlp("Grant Andersen")], on_match=add_name_ent) matches = matcher(doc) for ent in doc.ents: print((ent.text, ent.start, ent.end, ent.label_)) ``` -------------------------------- ### FuzzyMatcher Default Behavior (No Match) Source: https://github.com/gandersen101/spaczz/blob/main/README.md Illustrates the default fuzzy matching behavior where a name pattern does not match due to word order differences. This highlights the need for specific fuzzy matching configurations. ```python import spacy from spaczz.matcher import FuzzyMatcher nlp = spacy.blank("en") # Let's modify the order of the name in the text. text = """Anderson, Grint created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the US.""" # Spelling errors intentional. doc = nlp(text) matcher = FuzzyMatcher(nlp.vocab) matcher.add("NAME", [nlp("Grant Andersen")]) matches = matcher(doc) # The default fuzzy matching settings will not find a match. for match_id, start, end, ratio, pattern in matches: print(match_id, doc[start:end], ratio, pattern) ``` -------------------------------- ### SimilarityMatcher with Custom Settings Source: https://context7.com/gandersen101/spaczz/llms.txt Configures SimilarityMatcher with custom settings for minimum similarity (min_r2=60) and flexibility (flex=0) to match 'dog' with 'puppy' and 'hound'. ```python matcher2 = SimilarityMatcher(nlp.vocab) matcher2.add("ANIMAL", [nlp("dog")], kwargs=[{"min_r2": 60, "flex": 0}]) text2 = "The puppy and the hound played together." doc2 = nlp(text2) matches2 = matcher2(doc2) for match_id, start, end, ratio, pattern in matches2: print(f"Animal match: '{doc2[start:end]}', Similarity: {ratio}%") ``` -------------------------------- ### Checking FuzzyMatcher Properties Source: https://context7.com/gandersen101/spaczz/llms.txt Illustrates how to inspect the properties of a FuzzyMatcher instance, such as its labels, containment of specific labels, and the total number of patterns added. ```python print(f"Labels: {matcher.labels}") # ('NAME', 'GPE') print(f"Contains NAME: {'NAME' in matcher}") # True print(f"Pattern count: {len(matcher)}") # 2 ``` -------------------------------- ### Initialize SimilarityMatcher Source: https://github.com/gandersen101/spaczz/blob/main/README.md Initialize the SimilarityMatcher with a spaCy vocabulary and optionally set `min_r2` to control the minimum similarity ratio required for matches. A spaCy model with word vectors is necessary. ```python import spacy from spaczz.matcher import SimilarityMatcher nlp = spacy.load("en_core_web_md") text = "I like apples, grapes and bananas." doc = nlp(text) # lowering min_r2 from default of 75 to produce matches in this example matcher = SimilarityMatcher(nlp.vocab, min_r2=65) matcher.add("FRUIT", [nlp("fruit")]) matches = matcher(doc) ``` -------------------------------- ### Access SimilarityMatcher Patterns Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Illustrates adding a pattern with specific keyword arguments (ignore_case=False) and then accessing all stored patterns and their settings using the 'patterns' property. Note that the output format includes the label, pattern text, type, and kwargs. ```python import spacy from spaczz.matcher import SimilarityMatcher nlp = spacy.load("en_core_web_md") matcher = SimilarityMatcher(nlp.vocab) matcher.add("AUTHOR", [nlp("Kerouac")], [{"ignore_case": False}]) matcher.patterns == [ { "label": "AUTHOR", "pattern": "Kerouac", "type": "similarity", "kwargs": {"ignore_case": False} }, ] ``` -------------------------------- ### Basic FuzzyMatcher Usage Source: https://github.com/gandersen101/spaczz/blob/main/README.md Demonstrates the fundamental use of FuzzyMatcher to find approximate matches in text. It returns the fuzzy ratio and matched pattern along with standard match information. ```python import spacy from spaczz.matcher import FuzzyMatcher nlp = spacy.blank("en") text = """Grint M Anderson created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the US.""" # Spelling errors intentional. doc = nlp(text) matcher = FuzzyMatcher(nlp.vocab) matcher.add("NAME", [nlp("Grant Andersen")]) matcher.add("GPE", [nlp("Nashville")]) matches = matcher(doc) for match_id, start, end, ratio, pattern in matches: print(match_id, doc[start:end], ratio, pattern) ``` -------------------------------- ### Using on_match Callback for Entity Addition Source: https://context7.com/gandersen101/spaczz/llms.txt Demonstrates how to use the `on_match` callback with FuzzyMatcher to automatically add detected entities to the document's `ents`. ```python def add_entity(matcher, doc, i, matches): match_id, start, end, ratio, pattern = matches[i] entity = Span(doc, start, end, label="PERSON") doc.ents += (entity,) matcher3 = FuzzyMatcher(nlp.vocab) matcher3.add("PERSON", [nlp("Ridley Scott")], on_match=add_entity) doc3 = nlp("Rdley Scot directed Alien.") matcher3(doc3) for ent in doc3.ents: print(f"Entity: {ent.text}, Label: {ent.label_}") ``` -------------------------------- ### Load spaCy Model and Process Text Source: https://github.com/gandersen101/spaczz/blob/main/README.md Initializes a spaCy model and processes a given text to demonstrate basic Named Entity Recognition (NER). This serves as a baseline before applying custom rules. ```python import spacy from spaczz.pipeline import SpaczzRuler nlp = spacy.load("en_core_web_md") text = """Anderson, Grint created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the USA. Some of his favorite bands are Converg and Protet the Zero.""" # Spelling errors intentional. doc = nlp(text) for ent in doc.ents: print((ent.text, ent.start, ent.end, ent.label_)) ``` -------------------------------- ### Initialize spaCy and SimilarityMatcher Source: https://github.com/gandersen101/spaczz/blob/main/notebooks/README.ipynb Initializes a spaCy model with word vectors and the SimilarityMatcher. A model with word vectors is required for meaningful similarity results. ```python import spacy from spaczz.matcher import SimilarityMatcher nlp = spacy.load("en_core_web_md") text = "I like apples, grapes and bananas." doc = nlp(text) ``` -------------------------------- ### Access SimilarityMatcher Labels Source: https://github.com/gandersen101/spaczz/blob/main/docs/reference.md Demonstrates how to add a pattern with a label and then retrieve all labels currently stored in the SimilarityMatcher using the 'labels' property. ```python import spacy from spaczz.matcher import SimilarityMatcher nlp = spacy.load("en_core_web_md") matcher = SimilarityMatcher(nlp.vocab) matcher.add("AUTHOR", [nlp("Kerouac")]) matcher.labels ``` -------------------------------- ### SpaczzRuler Pipeline Component Initialization Source: https://context7.com/gandersen101/spaczz/llms.txt Initializes a spaCy blank English model and loads text to be processed by the SpaczzRuler pipeline component. ```python import spacy from spaczz.pipeline import SpaczzRuler nlp = spacy.blank("en") text = """Anderson, Grint created spaczz in his home at 555 Fake St, Apt 5 in Nashv1le, TN 55555-1234 in the USA. Some of his favorite bands are Converg and Protet the Zero.""" doc = nlp(text) ``` -------------------------------- ### TokenMatcher with Fuzzy and Regex Patterns Source: https://github.com/gandersen101/spaczz/blob/main/notebooks/README.ipynb Demonstrates adding multiple fuzzy and regex patterns to the TokenMatcher for matching text. Use when fuzzy matching capabilities beyond spaCy's Matcher are required. ```python import spacy from spaczz.match import TokenMatcher nlp = spacy.load("en_core_web_md") text = """The manager gave me SQL databesE acess so now I can acces the Sequal DB. My manager's name is Grfield""" doc = nlp(text) matcher = TokenMatcher(vocab=nlp.vocab) matcher.add( "DATA", [ [ {"TEXT": "SQL"}, {"LOWER": {"FREGEX": "(database){s<=1}"}}, {"LOWER": {"FUZZY": "access"}}, ], [{"TEXT": {"FUZZY": "Sequel"}, "POS": "PROPN"}, {"LOWER": "db"}], ], ) matcher.add("NAME", [[{"TEXT": {"FUZZY": "Garfield"}}]]) matches = matcher(doc) for match_id, start, end, ratio, pattern in matches: print(match_id, doc[start:end], ratio, pattern) ``` -------------------------------- ### Serialize and Deserialize SpaczzRuler Source: https://context7.com/gandersen101/spaczz/llms.txt Demonstrates saving and loading a SpaczzRuler to/from bytes, disk, and JSONL format. Ensure the spaCy pipeline is correctly initialized. ```python import spacy import tempfile from spaczz.pipeline import SpaczzRuler nlp = spacy.blank("en") ruler = SpaczzRuler(nlp) ruler.add_patterns([ {"label": "AUTHOR", "pattern": "Kerouac", "type": "fuzzy", "id": "BEAT"}, {"label": "GPE", "pattern": "Nashville", "type": "fuzzy"}, {"label": "ZIP", "pattern": "zip_codes", "type": "regex", "kwargs": {"predef": True}}, ]) # Serialize to bytes ruler_bytes = ruler.to_bytes() new_ruler = SpaczzRuler(nlp) new_ruler.from_bytes(ruler_bytes) print(f"Loaded from bytes: {len(new_ruler.patterns)} patterns") # Save to disk (creates directory with patterns.jsonl and cfg.json) with tempfile.TemporaryDirectory() as tmp_dir: ruler.to_disk(f"{tmp_dir}/ruler") # Load from disk loaded_ruler = SpaczzRuler(nlp) loaded_ruler.from_disk(f"{tmp_dir}/ruler") print(f"Loaded from disk: {loaded_ruler.patterns}") # Save as single JSONL file ruler.to_disk(f"{tmp_dir}/patterns.jsonl") # Load from JSONL jsonl_ruler = SpaczzRuler(nlp) jsonl_ruler.from_disk(f"{tmp_dir}/patterns.jsonl") print(f"Loaded from JSONL: {len(jsonl_ruler.patterns)} patterns") # Access entity IDs print(f"Entity IDs: {ruler.ent_ids}") # ('BEAT',) ```