### Install DAWG Package Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Install the DAWG package using pip. This command fetches and installs the latest version from PyPI. ```default pip install DAWG ``` -------------------------------- ### Install dawg2 Source: https://github.com/pymorphy2-fork/dawg/blob/master/README.md Use pip to install the dawg2 package. The imported name remains 'dawg'. ```bash pip install dawg2 ``` -------------------------------- ### Create and use a CompletionDAWG Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Extend DAWG with key completion by creating a `CompletionDAWG`. This allows retrieving all keys starting with a given prefix. Iterator versions are available for memory efficiency. ```python import dawg # Create a CompletionDAWG words = ['foo', 'bar', 'foobar', 'food', 'football'] d = dawg.CompletionDAWG(words) # Get all keys (sorted) print(d.keys()) # ['bar', 'foo', 'foobar', 'food', 'football'] # Get all keys starting with a prefix print(d.keys('foo')) # ['foo', 'foobar', 'food', 'football'] print(d.keys('foob')) # ['foobar'] print(d.keys('xyz')) # [] # Check if any keys exist with a given prefix print(d.has_keys_with_prefix('foo')) # True print(d.has_keys_with_prefix('xyz')) # False print(d.has_keys_with_prefix('')) # True (any keys exist) # Iterator version for large datasets for key in d.iterkeys('foo'): print(key) # Output: # foo # foobar # food # football # All prefix operations from DAWG are also available print(d.prefixes('football')) # ['foo', 'football'] ``` -------------------------------- ### Run All Tests with Tox Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Execute all tests for the project using the tox automation tool. Ensure tox is installed before running. ```bash $ tox ``` -------------------------------- ### Find Keys with Prefix in CompletionDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Retrieve all keys in a CompletionDAWG that start with a specified prefix. This method is efficient for prefix-based searches. ```python completion_dawg.keys(u'foo') [u'foo', u'foobar'] ``` -------------------------------- ### Iterate Keys with Prefix in CompletionDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Iterate over all keys in a CompletionDAWG that start with a specified prefix. This is a memory-efficient way to process matching keys. ```python for key in completion_dawg.iterkeys(u'foo'): print(key) ``` -------------------------------- ### Create and use an IntDAWG Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Map unicode keys to integer values using `IntDAWG`. Supports creation from dictionaries or iterables of tuples. Use `get` with a default value for safe lookups. Note that only non-negative integers up to 2^31-1 are supported. ```python import dawg # Create from dict word_frequencies = {'apple': 100, 'banana': 50, 'cherry': 75} d = dawg.IntDAWG(word_frequencies) # Create from iterable of tuples data = [('foo', 1), ('bar', 5), ('foobar', 3)] d = dawg.IntDAWG(data) # Get values print(d['foo']) # 1 print(d['bar']) # 5 print(d.get('foo')) # 1 print(d.get('xyz')) # None print(d.get('xyz', 0)) # 0 (default value) # Check membership print('foo' in d) # True print('xyz' in d) # False # Value range: 0 to 2^31-1 (non-negative integers only) d = dawg.IntDAWG({'large': 2**31 - 1}) print(d['large']) # 2147483647 # Negative values raise ValueError try: dawg.IntDAWG({'negative': -1}) except ValueError as e: print("Negative values not supported") ``` -------------------------------- ### Get Value with Default from BytesDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Safely retrieve a value from BytesDAWG using the `get` method, providing a default value if the key is not found. This avoids KeyErrors. ```python bytes_dawg.get(u'foo', None) ``` -------------------------------- ### DAWG Persistence: Save and Load Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Demonstrates how to save DAWG structures to files and load them back using various methods including memory mapping, streams, serialization, and pickling. ```python import dawg import pickle from io import BytesIO # Create a DAWG words = ['apple', 'banana', 'cherry'] d = dawg.CompletionDAWG(words) # Save to file (most memory-efficient loading) d.save('words.dawg') # Load from file (recommended - uses memory mapping) d2 = dawg.CompletionDAWG() d2.load('words.dawg') print(d2.keys()) # ['apple', 'banana', 'cherry'] # One-liner load with method chaining d3 = dawg.CompletionDAWG().load('words.dawg') # Write to stream buf = BytesIO() d.write(buf) # Read from stream (uses 3x more memory than load()) buf.seek(0) d4 = dawg.CompletionDAWG() d4.read(buf) # Serialize to bytes data = d.tobytes() d5 = dawg.CompletionDAWG().frombytes(data) # Pickling (uses 3x more memory than load()) pickled = pickle.dumps(d) d6 = pickle.loads(pickled) ``` -------------------------------- ### Create and use a basic DAWG Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Create a DAWG from a list of words for membership testing and prefix operations. Bytes are also supported for lookups. Use `input_is_sorted=True` for better performance if the input data is already sorted. ```python import dawg # Create a DAWG from a list of words words = ['foo', 'bar', 'foobar', 'baz', 'food'] d = dawg.DAWG(words) # Check if a key exists print('foo' in d) # True print('fo' in d) # False print(b'foo' in d) # True (bytes also supported) # Find all prefixes of a given key print(d.prefixes('foobarz')) # ['foo', 'foobar'] print(d.prefixes('bar')) # ['bar'] print(d.prefixes('xyz')) # [] # Iterator version for memory efficiency for prefix in d.iterprefixes('foobarz'): print(prefix) # Output: # foo # foobar # Binary version of prefixes print(d.b_prefixes(b'foobarz')) # [b'foo', b'foobar'] # Create from sorted data for better performance sorted_words = sorted(['apple', 'banana', 'cherry']) d_sorted = dawg.DAWG(sorted_words, input_is_sorted=True) ``` -------------------------------- ### Empty DAWG Handling Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Demonstrates the behavior of an empty `CompletionDAWG`, including checking its keys, prefix existence, and membership. ```python empty = dawg.CompletionDAWG([]) print(empty.keys()) # [] print(empty.has_keys_with_prefix('')) # False print('anything' in empty) # False ``` -------------------------------- ### Run Benchmarks with Tox Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Execute the project's benchmarks using tox with a specific configuration file. This command is used to measure performance. ```bash $ tox -c bench.ini ``` -------------------------------- ### Save and Load RecordDAWG Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Demonstrates how to create, save, and load a RecordDAWG with a specified format. The format string must match during construction and loading. ```python record_data = [('key', (1, 2, 3))] rd = dawg.RecordDAWG(">3H", record_data) rd.save('records.dawg') # Load RecordDAWG (format specified at construction) rd2 = dawg.RecordDAWG(">3H") rd2.load('records.dawg') print(rd2['key']) # [(1, 2, 3)] ``` -------------------------------- ### Create DAWG and CompletionDAWG Instances Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Initialize DAWG and CompletionDAWG objects with an iterable of unicode keys. These classes are suitable for fast and memory-efficient string storage without value assignment. ```python import dawg words = [u'foo', u'bar', u'foobar', u'foö', u'bör'] base_dawg = dawg.DAWG(words) completion_dawg = dawg.CompletionDAWG(words) ``` -------------------------------- ### Fuzzy Matching with Similar Keys Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Illustrates how to find keys similar to a query using compiled replacement rules for fuzzy matching. Supports multi-value replacements and works with BytesDAWG. ```python import dawg # Create DAWG with words containing special characters words = ['foo', 'foö', 'bar', 'bär', 'cafe', 'café'] d = dawg.DAWG(words) # Compile replacement rules (one-way: o -> ö) replaces = dawg.DAWG.compile_replaces({'o': 'ö', 'e': 'é'}) # Find similar keys print(d.similar_keys('foo', replaces)) # ['foo', 'foö'] print(d.similar_keys('bar', replaces)) # ['bar'] (a->ä not in replaces) print(d.similar_keys('cafe', replaces)) # ['cafe', 'café'] # Multi-value replacements (one character maps to multiple options) words_ru = ['ДЕРЕВНЯ', 'ДЕРЁВНЯ', 'ОЗЕРА', 'ОЗЁРА'] d_ru = dawg.DAWG(words_ru) # Е can match both Е and Ё replaces_ru = dawg.DAWG.compile_replaces({'Е': 'Ё'}) print(d_ru.similar_keys('ДЕРЕВНЯ', replaces_ru)) # ['ДЕРЕВНЯ', 'ДЕРЁВНЯ'] print(d_ru.similar_keys('ОЗЕРА', replaces_ru)) # ['ОЗЕРА', 'ОЗЁРА'] # Multiple replacement targets replaces_multi = dawg.DAWG.compile_replaces({'е': ['ё', 'ѣ']}) # Works with BytesDAWG and RecordDAWG too data = [('foo', b'v1'), ('foö', b'v2')] bd = dawg.BytesDAWG(data) replaces = dawg.DAWG.compile_replaces({'o': 'ö'}) # similar_items returns key-value pairs print(bd.similar_items('foo', replaces)) # [('foo', b'v1'), ('foö', b'v2')] # similar_item_values returns just values print(bd.similar_item_values('foo', replaces)) # [b'v1', b'v2'] ``` -------------------------------- ### Handle DAWG Build Errors Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Illustrates how to catch `dawg.Error` exceptions that occur during DAWG construction, such as null bytes in keys. ```python import dawg # dawg.Error for build errors try: # Null bytes in keys are not allowed dawg.DAWG([b'foo\x00bar']) except dawg.Error as e: print(f"Build error: {e}") ``` -------------------------------- ### Create and use an IntCompletionDAWG Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Combine integer mapping and key completion with `IntCompletionDAWG`. Supports value lookups, key completion, and retrieving items (key-value pairs). Iterator versions are available for efficient iteration. ```python import dawg # Create with key-value pairs data = [('foo', 3), ('bar', 3), ('foobar', 6), ('food', 4)] d = dawg.IntCompletionDAWG(data) # IntDAWG operations print(d['foo']) # 3 print(d['foobar']) # 6 print(d.get('xyz', 0)) # 0 # CompletionDAWG operations print(d.keys()) # ['bar', 'foo', 'foobar', 'food'] print(d.keys('foo')) # ['foo', 'foobar', 'food'] print(d.has_keys_with_prefix('fo')) # True # Get items (key-value pairs) print(d.items()) # [('bar', 3), ('foo', 3), ('foobar', 6), ('food', 4)] print(d.items('foo')) # [('foo', 3), ('foobar', 6), ('food', 4)] # Iterator version for key, value in d.iteritems('foo'): print(f"{key}: {value}") # Output: # foo: 3 # foobar: 6 # food: 4 ``` -------------------------------- ### Create and Query RecordDAWG Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Shows how to create a RecordDAWG for structured data using struct format strings. Demonstrates retrieving values and handling keys, including sorted output with big-endian format. ```python import dawg # Define struct format (3 unsigned shorts, big-endian) # See: https://docs.python.org/3/library/struct.html#format-strings fmt = ">3H" # Create with key-tuple pairs data = [ ('foo', (1, 2, 3)), ('bar', (10, 20, 30)), ('foo', (4, 5, 6)), # duplicate key with different value ('foobar', (100, 200, 300)) ] d = dawg.RecordDAWG(fmt, data) # Get values (sorted lexicographically when using big-endian) print(d['foo']) # [(1, 2, 3), (4, 5, 6)] print(d['bar']) # [(10, 20, 30)] print(d['foobar']) # [(100, 200, 300)] # Keys and items print(d.keys()) # ['bar', 'foo', 'foo', 'foobar'] print(d.keys('foo')) # ['foo', 'foo', 'foobar'] print(d.items()) # [('bar', (10, 20, 30)), ('foo', (1, 2, 3)), ...] # Prefix operations print(d.prefixes('foobar123')) # ['foo', 'foobar'] # Different format examples # Single integer d1 = dawg.RecordDAWG("=i", [('key', (-42,))]) # Float and double d2 = dawg.RecordDAWG("=fd", [('key', (3.14, 2.718281828))]) # Mixed types: int, unsigned short, char d3 = dawg.RecordDAWG("=iHc", [('key', (100, 500, b'A'))]) # Big-endian for meaningful sort order data = [('num', (3, 2, 256)), ('num', (3, 2, 1)), ('num', (3, 2, 3))] d_be = dawg.RecordDAWG(">3H", data) print(d_be['num']) # [(3, 2, 1), (3, 2, 3), (3, 2, 256)] - sorted! ``` -------------------------------- ### Create and Query BytesDAWG Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Demonstrates creating a BytesDAWG from key-value tuples and retrieving values. Handles duplicate keys and missing keys with default values. ```python import dawg data = [ ('key1', b'value1'), ('key2', b'value2'), ('key1', b'value3'), # duplicate key with different value ('foobar', b'data') ] d = dawg.BytesDAWG(data) # Get values (returns list even for single values) print(d['key1']) # [b'value1', b'value3'] print(d['key2']) # [b'value2'] print(d['foobar']) # [b'data'] # Get with default print(d.get('key1')) # [b'value1', b'value3'] print(d.get('missing')) # None print(d.get('missing', [])) # [] # KeyError for missing keys with [] try: d['missing'] except KeyError: print("Key not found") # Keys and items with optional prefix filter print(d.keys()) # ['foobar', 'key1', 'key1', 'key2'] print(d.keys('key')) # ['key1', 'key1', 'key2'] print(d.items()) # [('foobar', b'data'), ('key1', b'value1'), ...] # Prefix operations print(d.prefixes('key123')) # ['key1'] # Custom payload separator (for compatibility with DAWG < 0.5) old_dawg = dawg.BytesDAWG(payload_separator=b'\xff') # old_dawg.load('old_format.dawg') ``` -------------------------------- ### Handle File I/O Errors Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Demonstrates catching `IOError` exceptions that can occur during file operations like loading a non-existent DAWG file. ```python import dawg # IOError for file operations try: d = dawg.DAWG() d.load('nonexistent.dawg') except IOError as e: print(f"File error: {e}") ``` -------------------------------- ### Handle KeyError with .get() Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Shows how to handle `KeyError` when accessing keys that do not exist in a BytesDAWG or RecordDAWG. Using the `.get()` method provides a safe way to access keys, returning `None` or a default value if the key is not found. ```python d = dawg.BytesDAWG([('foo', b'bar')]) try: d['missing'] except KeyError: print("Use .get() for safe access") print(d.get('missing')) # None print(d.get('missing', [])) # [] ``` -------------------------------- ### Create IntDAWG Mapping Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Initialize an IntDAWG with a list of (unicode_key, integer_value) tuples. IntDAWG provides a faster alternative to RecordDAWG for simple integer mappings. ```default >>> data = [ (u'foo', 1), (u'bar', 2) ] >>> int_dawg = dawg.IntDAWG(data) ``` -------------------------------- ### Find Prefixes of a Key in DAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Identify all prefixes of a given key that are present in the DAWG. This is useful for analyzing key structures. ```python base_dawg.prefixes(u'foobarz') ``` -------------------------------- ### Load DAWG from a File Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Load a DAWG object from a file using the load method. This reconstructs the DAWG in memory. ```default >>> d = dawg.DAWG() >>> d.load('words.dawg') ``` -------------------------------- ### Find Similar Keys with Character Replacement Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Find keys in a DAWG that are similar to a given key, based on a character replacement table. This allows for fuzzy matching. ```python replaces = dawg.DAWG.compile_replaces({u'o': u'ö'}) base_dawg.similar_keys(u'foo', replaces) base_dawg.similar_keys(u'foö', replaces) base_dawg.similar_keys(u'bor', replaces) ``` -------------------------------- ### Demonstrate Lexicographical Ordering in RecordDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Illustrates how the endianness of numeric values in RecordDAWG affects the lexicographical ordering of items. Big-endian format ('>') ensures meaningful ordering for numeric values. ```default >>> data = [('foo', (3, 2, 256)), ('foo', (3, 2, 1)), ('foo', (3, 2, 3))] >>> d = RecordDAWG("3H", data) >>> d.items() [(u'foo', (3, 2, 256)), (u'foo', (3, 2, 1)), (u'foo', (3, 2, 3))] >>> d2 = RecordDAWG(">3H", data) >>> d2.items() [(u'foo', (3, 2, 1)), (u'foo', (3, 2, 3)), (u'foo', (3, 2, 256))] ``` -------------------------------- ### Iterate Prefixes of a Key in DAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Iterate over all prefixes of a given key that exist in the DAWG. This provides a memory-efficient way to access key prefixes. ```python for prefix in base_dawg.iterprefixes(u'foobarz'): print(prefix) ``` -------------------------------- ### Check for Keys with Prefix in CompletionDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Determine if a CompletionDAWG contains any keys that begin with the given prefix. This provides a boolean result for prefix existence. ```python completion_dawg.has_keys_with_prefix(u'foo') ``` -------------------------------- ### Handle Invalid File Format Errors Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Illustrates catching `IOError` when attempting to load a file with an invalid DAWG format. ```python import tempfile with tempfile.NamedTemporaryFile(mode='w', suffix='.dawg', delete=False) as f: f.write("invalid data") path = f.name try: dawg.DAWG().load(path) except IOError as e: print(f"Invalid format: {e}") ``` -------------------------------- ### Create RecordDAWG with Data Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Construct a RecordDAWG instance by providing a format string and an iterable of (unicode_key, value_tuple) pairs. The zip function can be used to combine keys and values. ```default >>> keys = [u'foo', u'bar', u'foobar', u'foo'] >>> values = [(1, 2, 3), (2, 1, 0), (3, 3, 3), (2, 1, 5)] >>> data = zip(keys, values) >>> record_dawg = RecordDAWG(format, data) ``` -------------------------------- ### Write DAWG to a Stream Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Save the contents of a DAWG object to a binary stream. Ensure the file is opened in binary write mode ('wb'). ```default >>> with open('words.dawg', 'wb') as f: ... d.write(f) ``` -------------------------------- ### Handle DAWG Separator Errors Source: https://context7.com/pymorphy2-fork/dawg/llms.txt Shows how to handle `dawg.Error` when an invalid payload separator is used, specifically when the separator character appears within a key. ```python data = [('foo', b'data')] dawg.BytesDAWG(data, payload_separator=b'f') # 'f' appears in key except dawg.Error as e: print(f"Separator error: {e}") ``` -------------------------------- ### Check Key Existence in DAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Verify if a key exists within a DAWG or CompletionDAWG instance using the 'in' operator. This operation is efficient for checking membership. ```python u'foo' in base_dawg u'baz' in completion_dawg ``` -------------------------------- ### Save DAWG to a File Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Persist a DAWG object to a file using the save method. This is a convenient way to store and later load the DAWG. ```default >>> d.save('words.dawg') ``` -------------------------------- ### Read RecordDAWG from a Stream Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Load a RecordDAWG object from a binary stream. This method requires the format string to be defined beforehand. ```default >>> d = dawg.RecordDAWG(format_string) >>> with open('words.record-dawg', 'rb') as f: ... d.read(f) ``` -------------------------------- ### Load Old BytesDAWG or RecordDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/CHANGES.md Use the `payload_separator` argument when loading older BytesDAWG or RecordDAWG files to maintain alphabetical ordering. ```python BytesDAWG(payload_separator=b'\xff').load('old.dawg') ``` -------------------------------- ### Create BytesDAWG Instance with Key-Value Pairs Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Initialize a BytesDAWG object with an iterable of (unicode_key, bytes_value) tuples. This class stores binary data associated with keys and handles duplicate keys by storing all unique values. ```python data = [(u'key1', b'value1'), (u'key2', b'value2'), (u'key1', b'value3')] bytes_dawg = dawg.BytesDAWG(data) ``` -------------------------------- ### Define RecordDAWG Format String Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Define the format string for packing and unpacking binary data in RecordDAWG. Consult Python's struct module documentation for format string specifications. ```default >>> format = ">HHH" ``` -------------------------------- ### Load BytesDAWG with Custom Separator Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Load a BytesDAWG from a file, specifying a custom payload separator. This is useful for compatibility with older DAWG versions or when the default separator (chr(1)) might conflict with key content. ```default >>> BytesDAWG(payload_separator=b'\xff').load('old.dawg') ``` -------------------------------- ### Pickle and Unpickle DAWG Object Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Serialize and deserialize a DAWG object using Python's pickle module. This allows for saving and loading DAWG states. ```default >>> import pickle >>> data = pickle.dumps(d) >>> d2 = pickle.loads(data) ``` -------------------------------- ### Retrieve Value from IntDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Access the integer value associated with a given unicode key in an IntDAWG. ```default >>> int_dawg[u'foo'] 1 ``` -------------------------------- ### Access RecordDAWG Values Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Retrieve values from a RecordDAWG. Multiple values can exist for the same key, returned as a list. ```default >>> record_dawg['foo'] [(1, 2, 3), (2, 1, 5)] >>> record_dawg['foobar'] [(3, 3, 3)] ``` -------------------------------- ### Retrieve Values from BytesDAWG Source: https://github.com/pymorphy2-fork/dawg/blob/master/docs/index.md Access the list of byte values associated with a given key in a BytesDAWG. For unique keys, a list with a single value is returned. Raises KeyError for missing keys. ```python bytes_dawg[u'key1'] bytes_dawg[u'key2'] ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.