PyMystem3

https://pythonhosted.org/pymystem3/
Admin
PyMystem3 is a Python wrapper for the Yandex Mystem 3.0 morphological analyzer that provides text...
Tokens:3,913
Snippets:30
Trust Score:9.5
Update:6 months ago
Show doc for...
Tokens:
Raw
### Installation Methods

Source: https://pythonhosted.org/pymystem3/genindex.html

Methods for managing the installation of the Mystem binary.

```APIDOC
## install() and autoinstall()

### Description
Functions to install or automatically install the Mystem binary.

### Method
Functions in module pymystem3.mystem
```

--------------------------------

### Installation Functions

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Functions for automatically installing or manually installing the Mystem binary.

```APIDOC
## autoinstall

### Description
Install mystem binary as :py:const:`~pymystem3.constants.MYSTEM_BIN`. Do nothing if already installed.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
#### Query Parameters
- **out** (sys.stderr) - Optional - Standard error stream to output installation messages.

## install

### Description
Install mystem binary as :py:const:`~pymystem3.constants.MYSTEM_BIN`. Overwrite if already installed.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
#### Query Parameters
- **out** (sys.stderr) - Optional - Standard error stream to output installation messages.

```

--------------------------------

### Mystem.start()

Source: https://pythonhosted.org/pymystem3/genindex.html

Starts the Mystem process.

```APIDOC
## start()

### Description
Initializes and starts the Mystem process.

### Method
Method of pymystem3.mystem.Mystem class
```

--------------------------------

### Install Mystem Binary

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Functions to manage the installation of the Mystem binary. autoinstall checks for existence before installing, while install forces an overwrite.

```python
def autoinstall(out=sys.stderr):
    """
    Install mystem binary as :py:const:`~pymystem3.constants.MYSTEM_BIN`.
    Do nothing if already installed.
    """

    if os.path.isfile(MYSTEM_BIN):
        return
    install(out)
```

```python
def install(out=sys.stderr):
    """
    Install mystem binary as :py:const:`~pymystem3.constants.MYSTEM_BIN`.
    Overwrite if already installed.
    """

    import requests
    import tempfile

    url = _get_tarball_url()

    print("Installing mystem to %s from %s" % (MYSTEM_BIN, url), file=out)

    if not os.path.isdir(MYSTEM_DIR):
        os.makedirs(MYSTEM_DIR)

    tmp_fd, tmp_path = tempfile.mkstemp()
    try:
        r = requests.get(url, stream=True)
        with os.fdopen(tmp_fd, 'wb') as fd:
            for chunk in r.iter_content(64 * 1024):
                fd.write(chunk)
            fd.flush()

        if url.endswith('.tar.gz'):
            import tarfile
            tar = tarfile.open(tmp_path)
            try:
                tar.extract(MYSTEM_EXE, MYSTEM_DIR)
            finally:
                tar.close()
        elif url.endswith('.zip'):
            import zipfile
            zip = zipfile.ZipFile(tmp_path)
            try:
                zip.extractall(MYSTEM_DIR)
            finally:
                zip.close()
        else:
            raise NotImplementedError("Could not install mystem from %s" % url)
    finally:
        os.unlink(tmp_path)
```

--------------------------------

### Internal: Start MyStem Process

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Sets up the subprocess for MyStem, configuring stdin, stdout, and non-blocking I/O.

```python
def _start_mystem(self):
        self._proc = subprocess.Popen([self._mystem_bin] + self._mystemargs,
                                      stdin=subprocess.PIPE,
                                      stdout=subprocess.PIPE,
                                      bufsize=0,
                                      close_fds=True if _POSIX else False)

        self._procin, self._procout = self._proc.stdin, self._proc.stdout
        self._procout_no = self._procout.fileno()
        _set_non_blocking(self._procout)
```

--------------------------------

### Binary Management

Source: https://pythonhosted.org/pymystem3/pymystem3.html

Methods for installing and managing the Mystem binary.

```APIDOC
## pymystem3.mystem.autoinstall

### Description
Installs the mystem binary if it is not already installed.

## pymystem3.mystem.install

### Description
Installs the mystem binary, overwriting any existing installation.
```

--------------------------------

### Start MyStem Binary

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

This method runs the MyStem binary. It is not mandatory to call it explicitly; it can be useful to avoid waiting for MyStem to load.

```APIDOC
## POST /start

### Description
Runs the MyStem binary. This is optional and can be used to pre-load the MyStem process.

### Method
POST

### Endpoint
/start

### Parameters
None

### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
Indicates that the MyStem binary has been successfully started.

#### Response Example
None
```

--------------------------------

### Start MyStem Binary

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Initiates the MyStem binary process. Use this to avoid waiting for MyStem to load on subsequent calls.

```python
def start(self):
        """
        Run mystem binary.

        .. note:: It is not mandatory to call it. Use it if you want to avoid waiting for mystem loads.
        """

        self._start_mystem()
```

--------------------------------

### Define mystem binary paths

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/constants.html

Sets the platform-specific binary name and default installation directory for the mystem executable.

```python
# -*- coding: utf-8 -*-

import os.path
import sys


_WIN = sys.platform.startswith('win')

#: Name of mystem binary
MYSTEM_EXE = "mystem.exe" if _WIN else "mystem"
#: Directory of mystem binary
MYSTEM_DIR = os.path.expanduser("~/.local/bin")
#: Full path to mystem binary
MYSTEM_BIN = os.path.join(MYSTEM_DIR, MYSTEM_EXE)
```

--------------------------------

### Analyze Implementation (Non-Pipeline Mode)

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Processes text in non-pipeline mode, writing to MyStem's stdin and using communicate() to get stdout. Resets the process after use.

```python
def _analyze_impl(self, text):
            if isinstance(text, unicode):
                text = text.encode('utf-8')

            if self._proc is None:
                self._start_mystem()

            self._procin.write(text)
            self._procin.write(_NL)

            out, _ = self._proc.communicate()
            self._proc = None
            try:
                obj = json.loads(out)
            except (IOError, ValueError):
                raise RuntimeError("Problem has been occured. Current state:\ntext:\n%s\nout:\n%s" % \
                                   (text, out))

            return obj
```

--------------------------------

### Environment Configuration

Source: https://pythonhosted.org/pymystem3/_sources/pymystem3.txt

Details on how to configure the Mystem binary path using environment variables.

```APIDOC
## Environment Variable: MYSTEM_BIN

### Description
Specifies the path to the Mystem binary. If this environment variable is not set, the package defaults to the path defined in `pymystem3.constants.MYSTEM_BIN`.
```

--------------------------------

### Initialize Mystem Class

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

The Mystem class wraps the binary for use in Python. It automatically triggers autoinstall if the binary path is not provided.

```python
class Mystem(object):

    """
    Wrap mystem binary to be able it use from Python.

    The two main methods you may use are the :py:meth:`__init__` initializer,
    and the :py:meth:`analyze` method to process your data and get mystem
    output results.

    :param  mystem_bin: path to mystem binary
    :type   mystem_bin: str
    :param  grammar_info: glue grammatical information for same lemmas in output.
    :type   grammar_info: bool
    :param  disambiguation: apply disambiguation
    :type   disambiguation: bool
    :param  entire_input: copy entire input to output
    :type   entire_input: bool

    .. note:: Default value of :py:attr:`mystem_bin` can be overwritted by :envvar:`MYSTEM_BIN`.
    """

    def __init__(self, mystem_bin=None, grammar_info=True, disambiguation=True, entire_input=True):
        self._mystem_bin = mystem_bin
        self._grammar_info = grammar_info
        self._disambiguation = disambiguation
        self._entire_input = entire_input
        self._procin = None
        self._procout = None
        self._procout_no = None
        self._proc = None

        if self._mystem_bin is None:
            self._mystem_bin = os.environ.get("MYSTEM_BIN", None)

        if self._mystem_bin is None:
            autoinstall()
            self._mystem_bin = MYSTEM_BIN

        self._mystemargs = ["--format", "json"]
```

--------------------------------

### Module Overview

Source: https://pythonhosted.org/pymystem3/_sources/pymystem3.txt

Overview of the available modules within the pymystem3 package.

```APIDOC
## Modules

### pymystem3.constants
Contains constant definitions used by the package, including default binary paths.

### pymystem3.metadata
Provides metadata information regarding the package.

### pymystem3.mystem
Contains the core logic for interacting with the Mystem morphological analyzer.

### pymystem3
Main package module providing top-level access to functionality.
```

--------------------------------

### Mystem Class Initialization

Source: https://pythonhosted.org/pymystem3/pymystem3.html

Initializes the Mystem wrapper to interact with the morphological analyzer binary.

```APIDOC
## Mystem.__init__

### Description
Initializes the Mystem wrapper to process text data.

### Parameters
- **mystem_bin** (str) - Optional - Path to mystem binary.
- **grammar_info** (bool) - Optional - Glue grammatical information for same lemmas in output.
- **disambiguation** (bool) - Optional - Apply disambiguation.
- **entire_input** (bool) - Optional - Copy entire input to output.
```

--------------------------------

### Mystem.lemmatize()

Source: https://pythonhosted.org/pymystem3/genindex.html

Returns the lemmas for the provided text.

```APIDOC
## lemmatize()

### Description
Extracts lemmas from the input text.

### Method
Method of pymystem3.mystem.Mystem class
```

--------------------------------

### Analyze Implementation (Pipeline Mode)

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Processes text in pipeline mode, writing to MyStem's stdin and reading from stdout. Handles non-blocking I/O and JSON parsing.

```python
def _analyze_impl(self, text):
            if isinstance(text, unicode):
                text = text.encode('utf-8')

            if self._proc is None:
                self._start_mystem()

            self._procin.write(text)
            self._procin.write(_NL)
            self._procin.flush()

            sio = StringIO()
            obj = None
            select.select([self._procout_no], [], [])
            while True:
                try:
                    out = self._procout.read()
                    sio.write(out)
                    obj = json.loads(sio.getvalue().decode('utf-8'))
                    break
                except (IOError, ValueError):
                    rd, _, _ = select.select([self._procout_no], [], [], 30)
                    if self._procout_no not in rd:
                        raise RuntimeError("Problem has been occured. Current state:\ntext:\n%s\nout:\n%s\nsio:\n%s" % \
                                           (text, out, sio.getvalue()))

            return obj
```

--------------------------------

### Mystem.analyze()

Source: https://pythonhosted.org/pymystem3/genindex.html

Performs morphological analysis on the provided text.

```APIDOC
## analyze()

### Description
Analyzes the input text using the Mystem engine.

### Method
Method of pymystem3.mystem.Mystem class
```

--------------------------------

### Mystem Class

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

The core class for interacting with the Mystem morphological analyzer.

```APIDOC
## Mystem Class

### Description
Wrap mystem binary to be able it use from Python. The two main methods you may use are the :py:meth:`__init__` initializer, and the :py:meth:`analyze` method to process your data and get mystem output results.

### Parameters
#### Path Parameters
- **mystem_bin** (str) - Optional - Path to mystem binary. Default value can be overwritten by :envvar:`MYSTEM_BIN`.
- **grammar_info** (bool) - Optional - Glue grammatical information for same lemmas in output. Defaults to True.
- **disambiguation** (bool) - Optional - Apply disambiguation. Defaults to True.
- **entire_input** (bool) - Optional - Copy entire input to output. Defaults to True.

### Methods
#### __init__
Initializes the Mystem object.

#### analyze
Processes input data and returns Mystem output results.

### Request Example
```python
from pymystem3 import Mystem
mystem = Mystem()
result = mystem.analyze("hello world")
print(result)
```

### Response
#### Success Response (200)
- **analysis** (list) - A list of analyzed word objects.

#### Response Example
```json
[
    {
        "analysis": [
            {
                "lex": "привет",
                "gr": "S,perf,1s,excl",
                "wt": 1
            }
        ],
        "text": "привет"
    },
    {
        "analysis": [
            {
                "lex": "мир",
                "gr": "S,inp,gen"
            }
        ],
        "text": "мир"
    }
]
```

```

--------------------------------

### Morphological Analysis Methods

Source: https://pythonhosted.org/pymystem3/pymystem3.html

Methods for performing morphological analysis and lemmatization on input text.

```APIDOC
## Mystem.analyze

### Description
Performs morphological analysis on the provided text.

### Parameters
- **text** (str) - Required - The text to analyze.

### Response
- **result** (dict) - The result of the morphology analysis.

## Mystem.lemmatize

### Description
Performs morphological analysis and returns a list of lemmas.

### Parameters
- **text** (str) - Required - The text to analyze.

### Response
- **lemmas** (list) - A list of lemmas extracted from the text.
```

--------------------------------

### Extract Lemma from Analysis Object

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Static method to extract the lemma from a MyStem analysis result object. Falls back to the original text if lemma is not found.

```python
@staticmethod
def _get_lemma(o):
        lemma = None
        try:
            lemma = o['analysis'][0]['lex']
        except (KeyError, IndexError):
            lemma = o['text'] if 'text' in o else None
        return lemma
```

--------------------------------

### Lemmatize Text

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Analyzes the input text and returns a list of lemmas (base forms of words).

```APIDOC
## POST /lemmatize

### Description
Performs morphology analysis on a text and returns a list of lemmas (base forms of words).

### Method
POST

### Endpoint
/lemmatize

### Parameters
#### Query Parameters
- **text** (str) - Required - The text to lemmatize.

### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **lemmas** (list) - A list of lemmas extracted from the input text.

#### Response Example
[
  "example",
  "word"
]
```

--------------------------------

### Analyze Text

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Performs a morphological analysis on the input text, returning a dictionary with analysis results for each word.

```APIDOC
## POST /analyze

### Description
Makes morphology analysis for a given text. It processes the text line by line and returns a list of analysis results.

### Method
POST

### Endpoint
/analyze

### Parameters
#### Query Parameters
- **text** (str) - Required - The text to analyze.

### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **analysis_results** (dict) - A dictionary containing the morphology analysis results for the input text.

#### Response Example
{
  "analysis_results": [
    {
      "text": "example",
      "analysis": [
        {
          "lex": "example",
          "gr": "S,nom,sg,ev"
        }
      ]
    }
  ]
}
```

--------------------------------

### Lemmatize Text

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Analyzes text morphology and returns a list of lemmas. Handles encoding for Python 2 compatibility.

```python
def lemmatize(self, text):
        """
        Make morphology analysis for a text and return list of lemmas.

        :param  text:   text to analyze
        :type   text:   str
        :returns:       list of lemmas
        :rtype:         list
        """

        need_encode = (sys.version_info[0] < 3 and isinstance(text, str))

        infos = self.analyze(text)
        lemmas = list(ifilter(None, imap(self._get_lemma, infos)))

        if need_encode is True:
            lemmas = [l.encode('utf-8') for l in lemmas]

        return lemmas
```

--------------------------------

### Analyze Text Morphology

Source: https://pythonhosted.org/pymystem3/_modules/pymystem3/mystem.html

Performs morphological analysis on a given text, splitting it into lines and processing each.

```python
def analyze(self, text):
        """
        Make morphology analysis for a text.

        :param  text:   text to analyze
        :type   text:   str
        :returns:       result of morphology analysis.
        :rtype:         dict
        """

        result = []
        for line in text.splitlines():
            result.extend(self._analyze_impl(line))
        return result
```
Tokens:
Raw