textract

https://textract.readthedocs.io/en/stable/
Admin
textract is a Python library that extracts text from various document formats including PDF, Word,...
Tokens:5,496
Snippets:42
Trust Score:8.3
Update:7 months ago
Show doc for...
Tokens:
Raw
### Basic Text Extraction with textract

Source: https://textract.readthedocs.io/en/stable/python_package

Demonstrates the fundamental usage of the textract.process function to extract text from a file. This is the most common way to get text content from documents.

```python
import textract
text = textract.process('path/to/file.extension')
```

--------------------------------

### Extract Text using Command Line Interface

Source: https://textract.readthedocs.io/en/stable/index

This command-line interface allows users to extract text from a specified file by providing its path. It's a straightforward way to get text content without writing any code. The output is the extracted text directly to the console.

```bash
textract path/to/file.extension
```

--------------------------------

### BaseParser: Process File with Encoding

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils

The `process` method in BaseParser orchestrates text extraction and encoding. It calls `extract` to get raw text, `decode` to handle potential byte strings, and `encode` to ensure the output matches the desired encoding.

```python
def process(self, filename, encoding, **kwargs):
        """Process ``filename`` and encode byte-string with ``encoding``. This
        method is called by :func:`textract.parsers.process` and wraps
        the :meth:`.BaseParser.extract` method in `a delicious unicode
        sandwich <http://nedbatchelder.com/text/unipain.html>`_.

        """
        # make a "unicode sandwich" to handle dealing with unknown
        # input byte strings and converting them to a predictable
        # output encoding
        # http://nedbatchelder.com/text/unipain/unipain.html#35
        byte_string = self.extract(filename, **kwargs)
        unicode_string = self.decode(byte_string)
        return self.encode(unicode_string, encoding)
```

--------------------------------

### Get Available File Extensions (Python)

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers

The `_get_available_extensions` function scans the `textract.parsers` directory for files ending with `_parser.py` to identify supported file types. It extracts the file extensions from these filenames and also includes extensions from the `EXTENSION_SYNONYMS` dictionary. This list is used for tab-completion and error handling, ensuring users know which file types are supported. The function returns a sorted list of extensions.

```python
def _get_available_extensions():
    """Get a list of available file extensions to make it easy for
    tab-completion and exception handling.
    """
    extensions = []

    # from filenames
    parsers_dir = os.path.join(os.path.dirname(__file__)))
    glob_filename = os.path.join(parsers_dir, "*" + _FILENAME_SUFFIX + ".py")
    ext_re = re.compile(glob_filename.replace('*', "(?P<ext>\w+)"))
    for filename in glob.glob(glob_filename):
        ext_match = ext_re.match(filename)
        ext = ext_match.groups()[0]
        extensions.append(ext)
        extensions.append('.' + ext)

    # from relevant synonyms (don't use the '' synonym)
    for ext in EXTENSION_SYNONYMS.keys():
        if ext:
            extensions.append(ext)
            extensions.append(ext.replace('.', '', 1))
    extensions.sort()
    return extensions

```

--------------------------------

### Advanced Text Extraction with Language and Method Options

Source: https://textract.readthedocs.io/en/stable/python_package

Demonstrates using advanced options like specifying the parsing method ('tesseract') and language ('nor') for OCR-ing text from a PDF file. This is useful for multilingual documents or when specific OCR engines are preferred.

```python
text = textract.process(
    'path/to/norwegian.pdf',
    method='tesseract',
    language='nor',
)
```

--------------------------------

### Text Extraction When File Has No Extension

Source: https://textract.readthedocs.io/en/stable/python_package

Explains how to extract text from a file that lacks a standard file extension by explicitly providing the extension as an argument to textract.process.

```python
import textract
text = textract.process('path/to/file', extension='docx')
```

--------------------------------

### Text Extraction with Specified Output Encoding

Source: https://textract.readthedocs.io/en/stable/python_package

Illustrates how to set a specific output encoding, like 'ascii', for the extracted text using textract.process. Input encodings are automatically detected using chardet.

```python
import textract
text = textract.process('path/to/file.extension', encoding='ascii')
```

--------------------------------

### Text Extraction with Specific PDF Parsing Method

Source: https://textract.readthedocs.io/en/stable/python_package

Shows how to specify a particular method for parsing PDF files, such as 'pdfminer', when using textract.process. This allows for customized PDF text extraction.

```python
import textract
text = textract.process('path/to/a.pdf', method='pdfminer')
```

--------------------------------

### Route Request to Appropriate Parser (Python)

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers

The `process` function in `textract.parsers` is the main entry point for text extraction. It takes a filename, encoding, and optional extension, validates the file's existence, determines the file type by its extension (handling synonyms), dynamically imports the corresponding parser module, and then calls the parser's `process` method to extract text. It raises `MissingFileError` if the file does not exist and `ExtensionNotSupported` if the file extension is not recognized.

```python
"""
Route the request to the appropriate parser based on file type.
"""

import os
import importlib
import glob
import re

from .. import exceptions

# Dictionary structure for synonymous file extension types
EXTENSION_SYNONYMS = {
    ".jpeg": ".jpg",
    ".tff": ".tiff",
    ".tif": ".tiff",
    ".htm": ".html",
    "": ".txt",
    ".log": ".txt",
}

# default encoding that is returned by the process method. specify it
# here so the default is used on both the process function and also by
# the command line interface
DEFAULT_ENCODING = 'utf_8'

# filename format
_FILENAME_SUFFIX = '_parser'



[docs]
def process(filename, encoding=DEFAULT_ENCODING, extension=None, **kwargs):
    """This is the core function used for extracting text. It routes the
    ``filename`` to the appropriate parser and returns the extracted
    text as a byte-string encoded with ``encoding``.
    """

    # make sure the filename exists
    if not os.path.exists(filename):
        raise exceptions.MissingFileError(filename)

    # get the filename extension, which is something like .docx for
    # example, and import the module dynamically using importlib. This
    # is a relative import so the name of the package is necessary
    # normally, file extension will be extracted from the file name
    # if the file name has no extension, then the user can pass the
    # extension as an argument
    if extension:
        ext = extension
        # check if the extension has the leading .
        if not ext.startswith('.'):
            ext = '.' + ext
        ext = ext.lower()
    else:
        _, ext = os.path.splitext(filename)
        ext = ext.lower()

    # check the EXTENSION_SYNONYMS dictionary
    ext = EXTENSION_SYNONYMS.get(ext, ext)

    # to avoid conflicts with packages that are installed globally
    # (e.g. python's json module), all extension parser modules have
    # the _parser extension
    rel_module = ext + _FILENAME_SUFFIX

    # If we can't import the module, the file extension isn't currently
    # supported
    try:
        filetype_module = importlib.import_module(
            rel_module, 'textract.parsers'
        )
    except ImportError:
        raise exceptions.ExtensionNotSupported(ext)

    # do the extraction

    parser = filetype_module.Parser()
    return parser.process(filename, encoding, **kwargs)

```

--------------------------------

### Extract Text using Python Package

Source: https://textract.readthedocs.io/en/stable/index

The textract Python package provides a programmatic way to extract text from files. By importing the 'textract' module and calling the 'process' function with a file path, developers can obtain the text content within their Python applications. The function returns the extracted text as a string.

```python
# some python file
import textract
text = textract.process("path/to/file.extension")
```

--------------------------------

### ShellParser: Run External Command

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils

The `run` method in ShellParser executes an external command specified by `args`. It captures stdout and stderr, returning them as a tuple. If the command fails, it raises a `ShellError`.

```python
def run(self, args):
        """Run ``command`` and return the subsequent ``stdout`` and ``stderr``
        as a tuple. If the command is not successful, this raises a
        :exc:`textract.exceptions.ShellError`.
        """

        # run a subprocess and put the stdout and stderr on the pipe object
        try:
            pipe = subprocess.Popen(
                args,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE,
            )
        except OSError as e:
            if e.errno == errno.ENOENT:
                # File not found.
                # This is equivalent to getting exitcode 127 from sh
                raise exceptions.ShellError(
                    ' '.join(args), 127, '', '',
                )

        # pipe.wait() ends up hanging on large files. using
        # pipe.communicate appears to avoid this issue
        stdout, stderr = pipe.communicate()

        # if pipe is busted, raise an error (unlike Fabric)
        if pipe.returncode != 0:
            raise exceptions.ShellError(
                ' '.join(args), pipe.returncode, stdout, stderr,
            )

        return stdout, stderr
```

--------------------------------

### BaseParser: Extract Text from File

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils

The BaseParser class provides a foundation for text extraction. Its `extract` method is intended to be overridden by subclasses to implement specific file parsing logic, returning raw text.

```python
class BaseParser(object):
    """The :class:`.BaseParser` abstracts out some common functionality
    that is used across all document Parsers. In particular, it has
    the responsibility of handling all unicode and byte-encoding.
    """


    def extract(self, filename, **kwargs):
        """This method must be overwritten by child classes to extract raw
        text from a filename. This method can return either a
        byte-encoded string or unicode.
        """
        raise NotImplementedError('must be overwritten by child classes')
```

--------------------------------

### Extract Text from DOC Files using antiword (Python)

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/doc_parser

This Python code defines a Parser class for extracting text from .doc files. It utilizes the 'antiword' command-line utility by inheriting from ShellParser and executing the 'antiword' command with the filename as an argument. The extracted standard output is returned.

```python
from .utils import ShellParser



[docs]class Parser(ShellParser):
    """Extract text from doc files using antiword.
    """


    [docs]
    def extract(self, filename, **kwargs):
        stdout, stderr = self.run(['antiword', filename])
        return stdout

```

--------------------------------

### BaseParser: Encode Text to Bytes

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils

The `encode` method within the BaseParser class handles the conversion of text to a specified byte encoding. It ignores characters that cannot be represented in the target encoding.

```python
def encode(self, text, encoding):
        """Encode the ``text`` in ``encoding`` byte-encoding. This ignores
        code points that can't be encoded in byte-strings.
        """
        return text.encode(encoding, 'ignore')
```

--------------------------------

### BaseParser: Decode Text with Chardet

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils

The `decode` method in BaseParser uses the `chardet` library to automatically detect and decode byte strings into Unicode. It handles cases where the input might already be Unicode or empty.

```python
def decode(self, text):
        """Decode ``text`` using the `chardet
        <https://github.com/chardet/chardet>`_ package.
        """
        # only decode byte strings into unicode if it hasn't already
        # been done by a subclass
        if isinstance(text, six.text_type):
            return text

        # empty text? nothing to decode
        if not text:
            return u''

        # use chardet to automatically detect the encoding text
        result = chardet.detect(text)
        return text.decode(result['encoding'])
```

--------------------------------

### ShellParser: Create Temporary Filename

Source: https://textract.readthedocs.io/en/stable/_modules/textract/parsers/utils

The `temp_filename` method in ShellParser generates a unique temporary filename using Python's `tempfile` module. It ensures the file handle is closed immediately after creation.

```python
def temp_filename(self):
        """Return a unique tempfile name.
        """
        # TODO: it would be nice to get this to behave more like a
        # context so we can make sure these temporary files are
        # removed, regardless of whether an error occurs or the
        # program is terminated.
        handle, filename = tempfile.mkstemp()
        os.close(handle)
        return filename
```
Tokens:
Raw