### Basic Text Extraction with textract Source: https://textract.readthedocs.io/en/stable/python_package Demonstrates the fundamental usage of the textract.process function to extract text from a file. This is the most common way to get text content from documents. ```python import textract text = textract.process('path/to/file.extension') ``` -------------------------------- ### Extract Text using Command Line Interface Source: https://textract.readthedocs.io/en/stable/index This command-line interface allows users to extract text from a specified file by providing its path. It's a straightforward way to get text content without writing any code. The output is the extracted text directly to the console. ```bash textract path/to/file.extension ``` -------------------------------- ### BaseParser: Process File with Encoding Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils The `process` method in BaseParser orchestrates text extraction and encoding. It calls `extract` to get raw text, `decode` to handle potential byte strings, and `encode` to ensure the output matches the desired encoding. ```python def process(self, filename, encoding, **kwargs): """Process ``filename`` and encode byte-string with ``encoding``. This method is called by :func:`textract.parsers.process` and wraps the :meth:`.BaseParser.extract` method in `a delicious unicode sandwich `_. """ # make a "unicode sandwich" to handle dealing with unknown # input byte strings and converting them to a predictable # output encoding # http://nedbatchelder.com/text/unipain/unipain.html#35 byte_string = self.extract(filename, **kwargs) unicode_string = self.decode(byte_string) return self.encode(unicode_string, encoding) ``` -------------------------------- ### Get Available File Extensions (Python) Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers The `_get_available_extensions` function scans the `textract.parsers` directory for files ending with `_parser.py` to identify supported file types. It extracts the file extensions from these filenames and also includes extensions from the `EXTENSION_SYNONYMS` dictionary. This list is used for tab-completion and error handling, ensuring users know which file types are supported. The function returns a sorted list of extensions. ```python def _get_available_extensions(): """Get a list of available file extensions to make it easy for tab-completion and exception handling. """ extensions = [] # from filenames parsers_dir = os.path.join(os.path.dirname(__file__))) glob_filename = os.path.join(parsers_dir, "*" + _FILENAME_SUFFIX + ".py") ext_re = re.compile(glob_filename.replace('*', "(?P\w+)")) for filename in glob.glob(glob_filename): ext_match = ext_re.match(filename) ext = ext_match.groups()[0] extensions.append(ext) extensions.append('.' + ext) # from relevant synonyms (don't use the '' synonym) for ext in EXTENSION_SYNONYMS.keys(): if ext: extensions.append(ext) extensions.append(ext.replace('.', '', 1)) extensions.sort() return extensions ``` -------------------------------- ### Advanced Text Extraction with Language and Method Options Source: https://textract.readthedocs.io/en/stable/python_package Demonstrates using advanced options like specifying the parsing method ('tesseract') and language ('nor') for OCR-ing text from a PDF file. This is useful for multilingual documents or when specific OCR engines are preferred. ```python text = textract.process( 'path/to/norwegian.pdf', method='tesseract', language='nor', ) ``` -------------------------------- ### Text Extraction When File Has No Extension Source: https://textract.readthedocs.io/en/stable/python_package Explains how to extract text from a file that lacks a standard file extension by explicitly providing the extension as an argument to textract.process. ```python import textract text = textract.process('path/to/file', extension='docx') ``` -------------------------------- ### Text Extraction with Specified Output Encoding Source: https://textract.readthedocs.io/en/stable/python_package Illustrates how to set a specific output encoding, like 'ascii', for the extracted text using textract.process. Input encodings are automatically detected using chardet. ```python import textract text = textract.process('path/to/file.extension', encoding='ascii') ``` -------------------------------- ### Text Extraction with Specific PDF Parsing Method Source: https://textract.readthedocs.io/en/stable/python_package Shows how to specify a particular method for parsing PDF files, such as 'pdfminer', when using textract.process. This allows for customized PDF text extraction. ```python import textract text = textract.process('path/to/a.pdf', method='pdfminer') ``` -------------------------------- ### Route Request to Appropriate Parser (Python) Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers The `process` function in `textract.parsers` is the main entry point for text extraction. It takes a filename, encoding, and optional extension, validates the file's existence, determines the file type by its extension (handling synonyms), dynamically imports the corresponding parser module, and then calls the parser's `process` method to extract text. It raises `MissingFileError` if the file does not exist and `ExtensionNotSupported` if the file extension is not recognized. ```python """ Route the request to the appropriate parser based on file type. """ import os import importlib import glob import re from .. import exceptions # Dictionary structure for synonymous file extension types EXTENSION_SYNONYMS = { ".jpeg": ".jpg", ".tff": ".tiff", ".tif": ".tiff", ".htm": ".html", "": ".txt", ".log": ".txt", } # default encoding that is returned by the process method. specify it # here so the default is used on both the process function and also by # the command line interface DEFAULT_ENCODING = 'utf_8' # filename format _FILENAME_SUFFIX = '_parser' [docs] def process(filename, encoding=DEFAULT_ENCODING, extension=None, **kwargs): """This is the core function used for extracting text. It routes the ``filename`` to the appropriate parser and returns the extracted text as a byte-string encoded with ``encoding``. """ # make sure the filename exists if not os.path.exists(filename): raise exceptions.MissingFileError(filename) # get the filename extension, which is something like .docx for # example, and import the module dynamically using importlib. This # is a relative import so the name of the package is necessary # normally, file extension will be extracted from the file name # if the file name has no extension, then the user can pass the # extension as an argument if extension: ext = extension # check if the extension has the leading . if not ext.startswith('.'): ext = '.' + ext ext = ext.lower() else: _, ext = os.path.splitext(filename) ext = ext.lower() # check the EXTENSION_SYNONYMS dictionary ext = EXTENSION_SYNONYMS.get(ext, ext) # to avoid conflicts with packages that are installed globally # (e.g. python's json module), all extension parser modules have # the _parser extension rel_module = ext + _FILENAME_SUFFIX # If we can't import the module, the file extension isn't currently # supported try: filetype_module = importlib.import_module( rel_module, 'textract.parsers' ) except ImportError: raise exceptions.ExtensionNotSupported(ext) # do the extraction parser = filetype_module.Parser() return parser.process(filename, encoding, **kwargs) ``` -------------------------------- ### Extract Text using Python Package Source: https://textract.readthedocs.io/en/stable/index The textract Python package provides a programmatic way to extract text from files. By importing the 'textract' module and calling the 'process' function with a file path, developers can obtain the text content within their Python applications. The function returns the extracted text as a string. ```python # some python file import textract text = textract.process("path/to/file.extension") ``` -------------------------------- ### ShellParser: Run External Command Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils The `run` method in ShellParser executes an external command specified by `args`. It captures stdout and stderr, returning them as a tuple. If the command fails, it raises a `ShellError`. ```python def run(self, args): """Run ``command`` and return the subsequent ``stdout`` and ``stderr`` as a tuple. If the command is not successful, this raises a :exc:`textract.exceptions.ShellError`. """ # run a subprocess and put the stdout and stderr on the pipe object try: pipe = subprocess.Popen( args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) except OSError as e: if e.errno == errno.ENOENT: # File not found. # This is equivalent to getting exitcode 127 from sh raise exceptions.ShellError( ' '.join(args), 127, '', '', ) # pipe.wait() ends up hanging on large files. using # pipe.communicate appears to avoid this issue stdout, stderr = pipe.communicate() # if pipe is busted, raise an error (unlike Fabric) if pipe.returncode != 0: raise exceptions.ShellError( ' '.join(args), pipe.returncode, stdout, stderr, ) return stdout, stderr ``` -------------------------------- ### BaseParser: Extract Text from File Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils The BaseParser class provides a foundation for text extraction. Its `extract` method is intended to be overridden by subclasses to implement specific file parsing logic, returning raw text. ```python class BaseParser(object): """The :class:`.BaseParser` abstracts out some common functionality that is used across all document Parsers. In particular, it has the responsibility of handling all unicode and byte-encoding. """ def extract(self, filename, **kwargs): """This method must be overwritten by child classes to extract raw text from a filename. This method can return either a byte-encoded string or unicode. """ raise NotImplementedError('must be overwritten by child classes') ``` -------------------------------- ### Extract Text from DOC Files using antiword (Python) Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/doc_parser This Python code defines a Parser class for extracting text from .doc files. It utilizes the 'antiword' command-line utility by inheriting from ShellParser and executing the 'antiword' command with the filename as an argument. The extracted standard output is returned. ```python from .utils import ShellParser [docs]class Parser(ShellParser): """Extract text from doc files using antiword. """ [docs] def extract(self, filename, **kwargs): stdout, stderr = self.run(['antiword', filename]) return stdout ``` -------------------------------- ### BaseParser: Encode Text to Bytes Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils The `encode` method within the BaseParser class handles the conversion of text to a specified byte encoding. It ignores characters that cannot be represented in the target encoding. ```python def encode(self, text, encoding): """Encode the ``text`` in ``encoding`` byte-encoding. This ignores code points that can't be encoded in byte-strings. """ return text.encode(encoding, 'ignore') ``` -------------------------------- ### BaseParser: Decode Text with Chardet Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils The `decode` method in BaseParser uses the `chardet` library to automatically detect and decode byte strings into Unicode. It handles cases where the input might already be Unicode or empty. ```python def decode(self, text): """Decode ``text`` using the `chardet `_ package. """ # only decode byte strings into unicode if it hasn't already # been done by a subclass if isinstance(text, six.text_type): return text # empty text? nothing to decode if not text: return u'' # use chardet to automatically detect the encoding text result = chardet.detect(text) return text.decode(result['encoding']) ``` -------------------------------- ### ShellParser: Create Temporary Filename Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils The `temp_filename` method in ShellParser generates a unique temporary filename using Python's `tempfile` module. It ensures the file handle is closed immediately after creation. ```python def temp_filename(self): """Return a unique tempfile name. """ # TODO: it would be nice to get this to behave more like a # context so we can make sure these temporary files are # removed, regardless of whether an error occurs or the # program is terminated. handle, filename = tempfile.mkstemp() os.close(handle) return filename ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.