### Install Documentation Dependencies

Source: https://github.com/ispras/dedoc/blob/master/docs/source/contributing/check_documentation.md

Install the necessary dependencies for building documentation using pip.

```bash
pip install .[docs]
```

--------------------------------

### Install and Set Up Pre-commit Hook

Source: https://github.com/ispras/dedoc/blob/master/docs/source/contributing/using_precommit.md

Install the pre-commit dependency and set up the git hook. This should be done once per repository.

```bash
pip3 install pre-commit
pre-commit install
```

--------------------------------

### Build and Install Tesseract OCR 5 from Source

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Adds a PPA for Tesseract development, updates package lists, installs Tesseract OCR and Russian language pack, clones Tesseract source, builds, and installs it. Sets the TESSDATA_PREFIX environment variable.

```bash
sudo add-apt-repository -y ppa:alex-p/tesseract-ocr-devel
sudo apt-get update --allow-releaseinfo-change
sudo apt-get install -y tesseract-ocr tesseract-ocr-rus
git clone --depth 1 --branch 5.0.0-beta-20210916 https://github.com/tesseract-ocr/tesseract/
cd tesseract && ./autogen.sh && sudo ./configure && sudo make && sudo make install && sudo ldconfig && cd ..
export TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata/
```

--------------------------------

### Install Dedoc Library with Pip (Basic)

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs the dedoc library using pip. Requires torch~=1.11.0 and torchvision~=0.12.0 to be already installed in the environment.

```bash
pip install dedoc
```

--------------------------------

### Install Dedoc Library

Source: https://github.com/ispras/dedoc/wiki/dedoc_lib_eng

Install Dedoc using pip within a virtual environment. Ensure you have Python 3 and have prepared your system environment by installing necessary packages, including LibreOffice.

```bash
virtualenv -p python3 . 
source bin/activate 
pip install -e git+https://github.com/ispras/dedoc@version_2021-02-18#egg=dedoc
```

--------------------------------

### Run Docker Compose from Labeling Directory

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_structure_type/dataset_creation.md

Alternatively, navigate to the 'labeling' directory and run Docker Compose to start the API. This is a more localized approach to starting the services.

```bash
cd labeling
docker-compose up --build
```

--------------------------------

### Install Flask for Task Manager

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_structure_type/dataset_creation.md

Installs the Flask web framework, which is required to run the task manager server. Ensure you are using Flask version 2.0.3.

```bash
pip3 install Flask==2.0.3
```

--------------------------------

### Example Document Tree Output

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/using_patterns.md

This is an example output of the document tree structure after applying custom patterns, showing how headers and list items are categorized.

```text
Document tree
**** id = 0 ; type = root

     **Title** id = 0.0 ; type = custom_header

     **Header 1** id = 0.1 ; type = custom_header

         **Header 1.1** id = 0.1.0 ; type = custom_header

             Text  id = 0.1.0.0 ; type = raw_text

             · bullet_list_item1  id = 0.1.0.1 ; type = custom_list

             ◦ subitem1  id = 0.1.0.2 ; type = custom_list

             ◦ subitem2  id = 0.1.0.3 ; type = custom_list

...

     **Header 2
    ********** id = 0.2 ; type = custom_header

         1\. Custom item
         id = 0.2.0 ; type = custom_list

             a) custom subitem
             id = 0.2.0.0 ; type = custom_list

         2\. Custom item 2
         id = 0.2.1 ; type = custom_list

         3\. Custom item 3
         id = 0.2.2 ; type = custom_list
...
```

--------------------------------

### Install dedoc Python Requirements and Launch Service

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Clones the dedoc repository, activates the virtual environment, sets the PYTHONPATH, installs project requirements including PyTorch, and launches the dedoc service.

```bash
# clone dedoc project
git clone https://github.com/ispras/dedoc.git
cd dedoc
# check on your's python environment
workon dedoc_env
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -r requirements.txt
pip install torch==1.11.0 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html
python dedoc/main.py -c ./dedoc/config.py
```

--------------------------------

### Install Required Packages for Trusted PyTorch

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs `mpich` and `intel-mkl` which are prerequisites for installing the trusted PyTorch version.

```bash
sudo apt-get install -y mpich intel-mkl
```

--------------------------------

### TagListPattern Usage Example (API)

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Example demonstrating how to use TagListPattern when interacting with the dedoc API.

```APIDOC
## API Usage Example

```python
import requests

patterns = [{"name": "tag_list", "line_type": "list_item", "default_level_1": 2, "can_be_multiline": "False"}]
parameters = {"patterns": str(patterns)}
with open(file_path, "rb") as file:
    files = {"file": (file_name, file)}
    r = requests.post("http://localhost:1231/upload", files=files, data=parameters)
```
```

--------------------------------

### Publisher and Date Node Example

Source: https://github.com/ispras/dedoc/blob/master/docs/source/structure_types/article.md

Example of 'publisher' and 'date' paragraph types within a document structure.

```json
{
  "node_id": "0.12.5.8",
  "text": "Springer",
  "annotations": [],
  "metadata": {
    "paragraph_type": "publisher",
    "page_id": 0,
    "line_id": 0
  },
  "subparagraphs": []
},
{
  "node_id": "0.12.5.9",
  "text": "2002",
  "annotations": [],
  "metadata": {
    "paragraph_type": "date",
    "page_id": 0,
    "line_id": 0
  },
  "subparagraphs": []
}
```

--------------------------------

### Install Tesseract OCR Dependencies

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs compilers and libraries required for building Tesseract OCR 5 from source. This is a prerequisite for OCR functionality.

```bash
sudo apt-get update
sudo apt-get install -y automake binutils-dev build-essential ca-certificates clang g++ g++-multilib gcc-multilib libcairo2 libffi-dev \
libgdk-pixbuf2.0-0 libglib2.0-dev libjpeg-dev libleptonica-dev libpango-1.0-0 libpango1.0-dev libpangocairo-1.0-0 libpng-dev libsm6 \
libtesseract-dev libtool make pkg-config poppler-utils pstotext shared-mime-info software-properties-common swig zlib1g-dev
```

--------------------------------

### Install Dedoc Library with Pip (Including Torch)

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs the dedoc library along with its torch and torchvision dependencies. Use this if torch and torchvision are not already present in your environment.

```bash
pip install "dedoc[torch]"
```

--------------------------------

### TagListPattern Usage Example (Library)

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Example demonstrating how to use TagListPattern within the dedoc library.

```APIDOC
## Library Usage Example

```python
import re
from dedoc.structure_extractors import DefaultStructureExtractor
from dedoc.structure_extractors.patterns import TagListPattern

reader = ...
structure_extractor = DefaultStructureExtractor()
patterns = [TagListPattern(line_type="list_item", default_level_1=2, can_be_multiline=False)]
document = reader.read(file_path=file_path)
document = structure_extractor.extract(document=document, parameters={"patterns": patterns})
```
```

--------------------------------

### Verify Dedoc Installation

Source: https://github.com/ispras/dedoc/wiki/dedoc_lib_eng

Verify the Dedoc installation by running a simple Python command to check if the get_unique_name utility function is accessible. This confirms the library is correctly installed and importable.

```python
python -c "from dedoc.utils import get_unique_name; print(get_unique_name('some.txt'))"
```

--------------------------------

### Instantiate DocxReader

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/usage.md

Create an instance of the DocxReader to parse .docx files. No setup is required beyond importing the class.

```python
reader = DocxReader()
```

--------------------------------

### Install Tesseract Language Package Locally

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_language.md

Install a specific Tesseract OCR language package, such as French (`fra`), on a local system using the `apt` package manager. This is required for Dedoc to perform OCR on documents in that language.

```bash
apt install -y tesseract-ocr-fra
```

--------------------------------

### Install System Dependencies for Dedoc

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs necessary system packages for dedoc converters (document formats to docx/xlsx/pptx) and archive extraction. Recommended for Ubuntu 20+.

```bash
sudo apt-get install -y libreoffice djvulibre-bin unzip unrar
```

--------------------------------

### Install Multiple Tesseract Language Packages Locally

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_language.md

Install Tesseract OCR language packages for multiple languages, such as French (`fra`) and Spanish (`spa`), using a loop and the `apt` package manager. This enables Dedoc to process documents in all specified languages.

```bash
export LANGUAGES="fra spa"
for lang in $LANGUAGES; do apt install -y tesseract-ocr-$lang; done
```

--------------------------------

### API Usage Example

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Example of how to use the Dedoc API for structure extraction with custom patterns.

```APIDOC
## POST /upload

### Description
Upload a file to the Dedoc API for document processing and structure extraction.

### Method
POST

### Endpoint
http://localhost:1231/upload

### Parameters
#### Query Parameters
- **patterns** (string) - Required - A JSON string representing the extraction patterns to apply.

#### Request Body
- **file** (file) - Required - The document file to be processed.

### Request Example
```python
import requests

patterns = [{"name": "tag", "default_line_type": "raw_text"}]
parameters = {"patterns": str(patterns)}
with open(file_path, "rb") as file:
    files = {"file": (file_name, file)}
    r = requests.post("http://localhost:1231/upload", files=files, data=parameters)
```

### Response
#### Success Response (200)
- **document** (object) - The processed document with extracted structure.

#### Response Example
```json
{
  "example": "response body"
}
```
```

--------------------------------

### Build and Run Docker Compose

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_structure_type/dataset_creation.md

Use Docker Compose to build the application images and start the services. This command is executed from the project root directory.

```bash
docker-compose -f labeling/docker-compose.yml up --build
```

--------------------------------

### Install Trusted PyTorch for Python 3.8

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs a verified version of PyTorch and Torchvision specifically for Python 3.8 from provided wheel URLs.

```bash
pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp38-cp38-linux_x86_64.whl
pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp38-cp38-linux_x86_64.whl
```

--------------------------------

### Initialize DocxConverter

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/usage.md

Instantiate the DocxConverter class to prepare for converting .odt files to .docx format. Ensure the dedoc library is installed.

```python
converter = DocxConverter()
```

--------------------------------

### Install Flake8 Requirements

Source: https://github.com/ispras/dedoc/blob/master/docs/source/contributing/using_flake8.md

Install the necessary requirements for the flake8 package to enable linting. This command should be run before using flake8.

```bash
pip3 install .[lint]
```

--------------------------------

### Library Usage Example

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Example of how to use Dedoc's DefaultStructureExtractor with custom TagPatterns in a Python script.

```APIDOC
## Library Usage

### Description
Demonstrates how to integrate Dedoc's structure extraction into a Python application using `DefaultStructureExtractor` and `TagPattern`.

### Code Example
```python
from dedoc.structure_extractors import DefaultStructureExtractor
from dedoc.structure_extractors.patterns import TagPattern

# Assume 'reader' is an initialized Dedoc reader object
reader = ... 

structure_extractor = DefaultStructureExtractor()
patterns = [TagPattern(default_line_type="raw_text")]
document = reader.read(file_path=file_path)
document = structure_extractor.extract(document=document, parameters={"patterns": patterns})
```
```

--------------------------------

### Install Trusted PyTorch for Python 3.9

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs a verified version of PyTorch and Torchvision specifically for Python 3.9 from provided wheel URLs.

```bash
pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torch-1.11.0a0+git137096a-cp39-cp39-linux_x86_64.whl
pip install https://github.com/ispras/dedockerfiles/raw/master/wheels/torchvision-0.12.0a0%2B9b5a3fe-cp39-cp39-linux_x86_64.whl
```

--------------------------------

### Set up Python Virtual Environment for dedoc

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/installation.md

Installs virtualenvwrapper and creates a dedicated virtual environment for dedoc. Ensure your Python version is compatible (3.8 or 3.9 recommended).

```bash
sudo pip3 install virtualenv virtualenvwrapper
mkdir ~/.virtualenvs
export WORKON_HOME=~/.virtualenvs
echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.8" >> ~/.bashrc
echo ". /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
source ~/.bashrc
mkvirtualenv dedoc_env
```

--------------------------------

### TagHeaderPattern Initialization and Usage

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Demonstrates how to initialize and use the TagHeaderPattern for structure extraction, including Python library usage and API examples.

```APIDOC
## TagHeaderPattern

### Description
Pattern for using information about heading lines (header) from readers saved in `line.metadata.tag_hierarchy_level`. Also allows to calculate `level_2` based on dotted list depth (same as in [`DottedListPattern`](#dedoc.structure_extractors.patterns.DottedListPattern)) **if level_2, tag_hierarchy_level.level_2, default_level_2 are empty**.

### Method
POST

### Endpoint
/upload

### Parameters
#### Query Parameters
- **patterns** (str) - Required - A JSON string representing the patterns to use.

#### Request Body
- **file** (file) - Required - The file to be processed.

### Request Example
```python
import requests

patterns = [{"name": "tag_header", "line_type": "header", "level_1": 1, "can_be_multiline": "False"}]
parameters = {"patterns": str(patterns)}
with open(file_path, "rb") as file:
    files = {"file": (file_name, file)}
    r = requests.post("http://localhost:1231/upload", files=files, data=parameters)
```

### Response
#### Success Response (200)
- **document** (Document) - The processed document with extracted structure.

#### Response Example
```json
{
  "content": [
    {
      "lines": [
        {
          "content": "Example Header",
          "metadata": {
            "tag_hierarchy_level": {
              "level_1": 1,
              "level_2": 0,
              "line_type": "header"
            }
          }
        }
      ]
    }
  ]
}
```

### SEE ALSO
Please see [Types of textual lines](../readers_output/line_types.md#readers-line-types) to find out which readers can extract lines with type “header”.

### *class* dedoc.structure_extractors.patterns.TagHeaderPattern(line_type: str | None = None, level_1: int | None = None, level_2: int | None = None, can_be_multiline: bool | str | None = None, default_line_type: str = 'header', default_level_1: int = 1, default_level_2: int | None = None)

Bases: [`TagPattern`](#dedoc.structure_extractors.patterns.TagPattern)

#### name = 'tag_header'

#### __init__(line_type: str | None = None, level_1: int | None = None, level_2: int | None = None, can_be_multiline: bool | str | None = None, default_line_type: str = 'header', default_level_1: int = 1, default_level_2: int | None = None) → None

Initialize pattern for configuring values of [`HierarchyLevel`](data_structures.md#dedoc.data_structures.HierarchyLevel) attributes. It is recommended to configure `default_*` values in case `line.metadata.tag_hierarchy_level` miss some values. If you want to use values from `line.metadata.tag_hierarchy_level`, it is recommended to leave `line_type`, `level_1`, `level_2`, `can_be_multiline` empty.

`can_be_multiline` is filled in PDF and images readers during paragraph detection, so if you want to extract paragraphs, you shouldn’t set `can_be_multiline` during pattern initialization.

* **Parameters:**
  * **line_type** – type of the line, replaces line_type from tag_hierarchy_level if non-empty.
  * **level_1** – value of a line primary importance, replaces level_1 from tag_hierarchy_level if non-empty.
  * **level_2** – level of the line inside specific class, replaces level_2 from tag_hierarchy_level if non-empty.
  * **can_be_multiline** – is used to unify lines inside tree node by [`TreeConstructor`](structure_constructors.md#dedoc.structure_constructors.TreeConstructor), if line can be multiline, it can be joined with another line. If not None, replaces can_be_multiline from tag_hierarchy_level.
  * **default_line_type** – type of the line, is used when tag_hierarchy_level.line_type == “unknown”.
  * **default_level_1** – value of a line primary importance, is used when tag_hierarchy_level.level_1 is None.
  * **default_level_2** – level of the line inside specific class, is used when tag_hierarchy_level.level_2 is None.
```

--------------------------------

### Run Docker Compose Services

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_language.md

Start the Dedoc services defined in `docker-compose.yml`, ensuring that the Docker images are built with the specified language configurations.

```bash
docker-compose up --build
```

--------------------------------

### Run Docker Container for Labeling

Source: https://github.com/ispras/dedoc/blob/master/labeling/resources/img_classifier_dockerfile/README.md

Runs the Docker container, mapping port 5555 and making it interactive. This command starts the labeling service, accessible via localhost:5555.

```bash
docker run -ti --rm -p 5555:5555 labeling
```

--------------------------------

### Run dedoc as a web application

Source: https://github.com/ispras/dedoc/blob/master/docs/source/dedoc_api_usage/api.md

To run the dedoc application after installing it via pip, use this command in your terminal.

```bash
dedoc -m main
```

--------------------------------

### Run Dedoc Task Manager Server

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_structure_type/dataset_creation.md

Starts the task manager server for distributing and collecting annotation tasks. Access the manager interface via localhost:3000.

```bash
python3 task_manager.py
```

--------------------------------

### Run Dedoc with Docker Compose on GPU

Source: https://github.com/ispras/dedoc/blob/master/docker_gpu/README.md

Execute this command from the `docker_gpu` directory to build and start Dedoc with GPU support. Ensure `on_gpu` is set to `True` in `config.py`.

```shell
cd docker_gpu
docker-compose up --build
```

--------------------------------

### Bibliography Reference Annotation Example

Source: https://github.com/ispras/dedoc/blob/master/docs/source/structure_types/article.md

Demonstrates a 'raw_text' node containing 'reference' annotations that link to bibliography items via UUIDs.

```json
{
  "node_id": "0.10.0",
  "text": "The results in this work essentially show that masking and leakage-resilient constructions hardly combine constructively. For (stateful) PRGs, our experiments indicate that both for software and hardware implementations, a leakageresilient design instantiated with an unprotected AES is the most efficient solution to reach any given security level. For stateless PRFs, they rather show that a bounded data complexity guarantee is (mostly) ineffective in bounding the (computational) complexity of the best attacks. So implementing masking and limiting the lifetime of the cryptographic implementation is the best solution in this case. Nevertheless, the chosen-plaintext tweak proposed in [34] is an interesting exception to this conclusion, as it leads to security-bounded hardware implementations for stateless primitives that are particularly interesting from an application point-of-view, e.g. for re-synchronization, challenge-response protocols, . . . Beyond the further analysis of such constructions, their extension to software implementations is an interesting scope for further research. In this respect, the combination of a chosen-plaintext leakage-resilient PRF with the shuffling countermeasure in [62] seems promising, as it could \"emulate\" the keydependent algorithmic noise ensuring security bounds in hardware.           ",
  "annotations": [
    {
      "start": 690,
      "end": 694,
      "name": "reference",
      "value": "109c08ee-0872-11ef-b95c-0242ac120002"
    },
    {
      "start": 1214,
      "end": 1218,
      "name": "reference",
      "value": "10a004ee-0872-11ef-b95c-0242ac120002"
    }
  ],
  "metadata": {
    "paragraph_type": "raw_text",
    "page_id": 0,
    "line_id": 0
  },
  "subparagraphs": []
}
```

--------------------------------

### Build HTML Documentation with Sphinx

Source: https://github.com/ispras/dedoc/blob/master/docs/source/contributing/check_documentation.md

Build the project documentation into HTML pages using Sphinx. Ensure you are in the project root directory.

```bash
python -m sphinx -T -E -W -b html -d docs/_build/doctrees -D language=en docs/source docs/_build
```

--------------------------------

### API Usage for StartWordPattern with requests

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

This example shows how to configure `StartWordPattern` via API call using the `requests` library. The `patterns` parameter should be a JSON string representation of the pattern configuration. Note that `can_be_multiline` is passed as a string 'false'.

```python
import requests

patterns = [{"name": "start_word", "start_word": "chapter", "line_type": "chapter", "level_1": 1, "can_be_multiline": "false"}]
parameters = {"patterns": str(patterns)}
with open(file_path, "rb") as file:
    files = {"file": (file_name, file)}
    r = requests.post("http://localhost:1231/upload", files=files, data=parameters)
```

--------------------------------

### Bibliography Item Node Example

Source: https://github.com/ispras/dedoc/blob/master/docs/source/structure_types/article.md

Example of a 'bibliography_item' node, which is referenced by 'reference' annotations in the text. It contains a unique 'uid'.

```json
{
  "node_id": "0.12.33",
  "text": "",
  "annotations": [],
  "metadata": {
    "paragraph_type": "bibliography_item",
    "page_id": 0,
    "line_id": 0,
    "uid": "109c08ee-0872-11ef-b95c-0242ac120002"
  }
}
```

```json
{
  "node_id": "0.12.61",
  "text": "",
  "annotations": [],
  "metadata": {
    "paragraph_type": "bibliography_item",
    "page_id": 0,
    "line_id": 0,
    "uid": "10a004ee-0872-11ef-b95c-0242ac120002"
  }
}
```

--------------------------------

### Use DjvuConverter in Code

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_doc_format.md

Demonstrates how to instantiate and use the DjvuConverter to check if a file can be converted and to perform the conversion from DJVU to PDF.

```python
file_path = "test_dir/The_New_Yorker_Case_Study.djvu"

djvu_converter = DjvuConverter()
djvu_converter.can_convert(file_path)  # True
djvu_converter.convert(file_path)  # 'test_dir/The_New_Yorker_Case_Study.pdf'
```

--------------------------------

### Build and Tag Docker Image for Multilingual Support

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_language.md

Build a Docker image with a specific tag and include support for multiple languages like French and Spanish using the LANGUAGES build argument.

```bash
docker build -t dedocproject/dedoc_multilang:latest --build-arg LANGUAGES="fra spa" .
```

--------------------------------

### Build Docker Container for Labeling

Source: https://github.com/ispras/dedoc/blob/master/labeling/resources/img_classifier_dockerfile/README.md

Builds the Docker image for the labeling application. Ensure you are in the task directory after unpacking the archive.

```bash
docker build -t labeling .
```

--------------------------------

### Initialize PatternComposition with Tag Patterns

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Demonstrates initializing PatternComposition with TagListPattern and TagPattern for defining line hierarchy. Ensure patterns are ordered from most specific to least specific.

```python
from dedoc.data_structures.line_with_meta import LineWithMeta
from dedoc.structure_extractors.patterns import TagListPattern, TagPattern
from dedoc.structure_extractors.patterns.pattern_composition import PatternComposition


pattern_composition = PatternComposition(
    patterns=[
        TagListPattern(line_type="list_item", default_level_1=2, can_be_multiline=False),
        TagPattern(default_line_type="raw_text")
    ]
)
line = LineWithMeta(line="Some text")
line.metadata.hierarchy_level = pattern_composition.get_hierarchy_level(line=line)
```

--------------------------------

### Default Structure Extractor Example

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_structure_type.md

This example demonstrates a structure extractor for the default structure type in Dedoc, utilizing regular expressions and heuristic rules. It serves as a reference for implementing custom heuristic-based extractors.

```python
class DefaultStructureExtractor(BaseStructureExtractor):
    def __init__(self, config: Optional[dict] = None):
        config = config or {}
        self.structure_type = config.get("structure_type", StructureType.DEFAULT)
        self.regexps = config.get("regexps", [
            re.compile(r"^([IVXLCDM]+)\."),  # Roman numerals
            re.compile(r"^(\d+)\."),  # Arabic numerals
            re.compile(r"^([a-zA-Zа-яА-Я])\."),  # Letters
        ])
        self.assume_table_rows = config.get("assume_table_rows", False)

    def extract(self, doc: Document, content: List[LineWithMeta]) -> Document:
        for line in content:
            if line.meta.structure_type is None:
                line.meta.structure_type = self.structure_type
            if line.meta.hierarchy_level is None:
                line.meta.hierarchy_level = HierarchyLevel.root

            for regexp in self.regexps:
                match = regexp.match(line.text)
                if match:
                    level = len(match.groups()[0])
                    line.meta.hierarchy_level = HierarchyLevel(level=level)
                    break

            if self.assume_table_rows and line.meta.table_type == TableType.table_content:
                line.meta.hierarchy_level = HierarchyLevel.table_row

        return doc
```

--------------------------------

### Run Labeling Application Container

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_structure_type/dataset_creation.md

Launches the Docker container for the image classification labeling application. Access the labeling interface at localhost:5555.

```bash
bash run.sh
```

--------------------------------

### LetterListPattern API Usage

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Example of how to use the LetterListPattern in an API request to extract documents with letter-based lists.

```APIDOC
## POST /upload

### Description
Upload a file for document processing with custom structure extraction patterns.

### Method
POST

### Endpoint
/upload

### Parameters
#### Query Parameters
- **patterns** (string) - Required - A JSON string representing the list of patterns to use for structure extraction.

#### Request Body
- **file** (file) - Required - The document file to be processed.

### Request Example
```python
import requests

file_path = "path/to/your/document.txt"
file_name = "document.txt"

patterns = [{"name": "letter_list", "line_type": "list_item", "level_1": 1, "level_2": 1, "can_be_multiline": "false"}]
parameters = {"patterns": str(patterns)}

with open(file_path, "rb") as file:
    files = {"file": (file_name, file)}
    r = requests.post("http://localhost:1231/upload", files=files, data=parameters)

print(r.json())
```

### Response
#### Success Response (200)
- **document** (object) - The processed document structure.

#### Response Example
```json
{
  "document": {
    "content": [
      {
        "lines": [
          {
            "content": "a) first element",
            "metadata": {
              "paragraph_type": "list_item",
              "hierarchy_level": {
                "line_type": "list_item",
                "level_1": 1,
                "level_2": 1,
                "can_be_multiline": false
              }
            }
          }
        ]
      }
    ]
  }
}
```
```

--------------------------------

### Using PdfReader

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_doc_format.md

Demonstrates how to instantiate and use the PdfReader class to read a PDF file and its attachments. It shows how to check if a file can be read and how to retrieve the UnstructuredDocument object.

```python
pdf_reader = PdfReader()

file_path = "test_dir/pdf_with_attachment.pdf"
pdf_reader.can_read(file_path)  # True
pdf_reader.read(file_path, parameters={"with_attachments": "true"})  # <dedoc.data_structures.UnstructuredDocument>

document = pdf_reader.read(file_path, parameters={"with_attachments": "true"})
list(vars(document))  # ['tables', 'lines', 'attachments', 'warnings', 'metadata']
len(document.attachments)  # 1
len(document.lines)  # 11
```

--------------------------------

### Get Hierarchy Level

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Determines the HierarchyLevel for a given line, applying configured pattern values or defaults.

```APIDOC
## get_hierarchy_level(line: LineWithMeta) -> HierarchyLevel

### Description
This method should only be called if the [`match()`](#dedoc.structure_extractors.patterns.TagPattern.match) method returned `True` for the given line. It returns a [`HierarchyLevel`](data_structures.md#dedoc.data_structures.HierarchyLevel) object for initializing `line.metadata.hierarchy_level`.

The `line_type` attribute is initialized based on these rules:
* If a non-empty `line_type` was provided during pattern initialization, its value is used.
* If `line_type` is `None` and `line.metadata.tag_hierarchy_level` is not 'unknown', the `line_type` from `line.metadata.tag_hierarchy_level` is used.
* Otherwise (if `line_type` is empty and `line.metadata.tag_hierarchy_level` is 'unknown'), the `default_line_type` value is used.

Similar rules apply for `level_1` and `level_2`, comparing against `None` instead of 'unknown'.

The `can_be_multiline` attribute is initialized as follows:
* If a non-empty `can_be_multiline` was provided during pattern initialization, its value is used.
* Otherwise, the `can_be_multiline` value from `line.metadata.tag_hierarchy_level` is used.

### Parameters
* **line** (LineWithMeta) - The line object to process.
```

--------------------------------

### Build and Run Dedoc with Docker Compose

Source: https://github.com/ispras/dedoc/blob/master/README.md

Build the Dedoc Docker image and start the application using Docker Compose. This command is used when you have cloned the repository and want to run Dedoc with custom configurations.

```shell
docker compose up --build
```

--------------------------------

### Use TagHeaderPattern in DefaultStructureExtractor

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Example of integrating TagHeaderPattern into DefaultStructureExtractor for processing documents. Requires a reader and file path.

```python
import re
from dedoc.structure_extractors import DefaultStructureExtractor
from dedoc.structure_extractors.patterns import TagHeaderPattern

reader = ...
structure_extractor = DefaultStructureExtractor()
patterns = [TagHeaderPattern(line_type="header", level_1=1, can_be_multiline=False)]
document = reader.read(file_path=file_path)
document = structure_extractor.extract(document=document, parameters={"patterns": patterns})
```

--------------------------------

### RomanListPattern API Usage

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

Example of how to use the RomanListPattern in an API request to extract documents with Roman numeral-based lists.

```APIDOC
## POST /upload

### Description
Upload a file for document processing with custom structure extraction patterns.

### Method
POST

### Endpoint
/upload

### Parameters
#### Query Parameters
- **patterns** (string) - Required - A JSON string representing the list of patterns to use for structure extraction.

#### Request Body
- **file** (file) - Required - The document file to be processed.

### Request Example
```python
import requests

file_path = "path/to/your/document.txt"
file_name = "document.txt"

patterns = [{"name": "roman_list", "line_type": "list_item", "level_1": 1, "level_2": 1, "can_be_multiline": "false"}]
parameters = {"patterns": str(patterns)}

with open(file_path, "rb") as file:
    files = {"file": (file_name, file)}
    r = requests.post("http://localhost:1231/upload", files=files, data=parameters)

print(r.json())
```

### Response
#### Success Response (200)
- **document** (object) - The processed document structure.

#### Response Example
```json
{
  "document": {
    "content": [
      {
        "lines": [
          {
            "content": "I. first item",
            "metadata": {
              "paragraph_type": "list_item",
              "hierarchy_level": {
                "line_type": "list_item",
                "level_1": 1,
                "level_2": 1,
                "can_be_multiline": false
              }
            }
          }
        ]
      }
    ]
  }
}
```
```

--------------------------------

### BracketRomanListPattern API

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

This section details the BracketRomanListPattern for matching roman lists with brackets and provides an example of its API usage.

```APIDOC
## POST /upload

### Description
Upload a file with custom structure extraction patterns.

### Method
POST

### Endpoint
/upload

### Parameters
#### Query Parameters
- **patterns** (string) - Required - A JSON string representing the list of patterns to use for structure extraction.

#### Request Body
- **file** (file) - Required - The file to be processed.

### Request Example
```json
{
  "patterns": "[{\"name\": \"bracket_roman_list\", \"line_type\": \"list_item\", \"level_1\": 1, \"level_2\": 1, \"can_be_multiline\": \"false\"}]"
}
```

### Response
#### Success Response (200)
- **document** (object) - The processed document with extracted structure.

#### Response Example
```json
{
  "document": { ... }
}
```
```

--------------------------------

### Initialize DocxMetadataExtractor

Source: https://github.com/ispras/dedoc/blob/master/docs/source/getting_started/usage.md

Instantiate the DocxMetadataExtractor to begin extracting metadata from DOCX files. Ensure the necessary reader has already processed the document.

```python
metadata_extractor = DocxMetadataExtractor()
```

--------------------------------

### BracketListPattern API

Source: https://github.com/ispras/dedoc/blob/master/docs/source/modules/structure_extractors.md

This section details the BracketListPattern for matching numbered lists with brackets and provides an example of its API usage.

```APIDOC
## POST /upload

### Description
Upload a file with custom structure extraction patterns.

### Method
POST

### Endpoint
/upload

### Parameters
#### Query Parameters
- **patterns** (string) - Required - A JSON string representing the list of patterns to use for structure extraction.

#### Request Body
- **file** (file) - Required - The file to be processed.

### Request Example
```json
{
  "patterns": "[{\"name\": \"bracket_list\", \"line_type\": \"list_item\", \"level_1\": 1, \"level_2\": 1, \"can_be_multiline\": \"false\"}]"
}
```

### Response
#### Success Response (200)
- **document** (object) - The processed document with extracted structure.

#### Response Example
```json
{
  "document": { ... }
}
```
```

--------------------------------

### Implement DjvuConverter Methods

Source: https://github.com/ispras/dedoc/wiki/adding_new_doc_type_tutotial_eng

Implement can_convert to check file extensions and do_convert to perform the file conversion using external utilities like ddjvu. Ensure the converted file path is returned.

```python
import os
from typing import Optional

from dedoc.converters.concrete_converters.abstract_converter import AbstractConverter


class DjvuConverter(AbstractConverter):
    def __init__(self, config):
        super().__init__(config=config)

    def can_convert(self, 
                    extension: str, 
                    mime: str, 
                    parameters: Optional[dict] = None) -> bool:
        return extension == '.djvu'

    def do_convert(self, 
                   tmp_dir: str, 
                   filename: str, 
                   extension: str) -> str:
        os.system("ddjvu -format=pdf "
                  "{tmp_dir}/{filename}{extension} {tmp_dir}/{filename}.pdf".format(tmp_dir=tmp_dir, 
                                                                                    filename=filename, 
                                                                                    extension=extension))
        self._await_for_conversion(filename + '.pdf', tmp_dir)
        return filename + '.pdf'
```

--------------------------------

### Clone Dedoc Repository

Source: https://github.com/ispras/dedoc/blob/master/docs/source/tutorials/add_new_language.md

Clone the dedoc repository and navigate to the dedoc directory to begin configuration.

```bash
git clone https://github.com/ispras/dedoc
cd dedoc
```

--------------------------------

### JSON Structure Example (Attachments)

Source: https://github.com/ispras/dedoc/blob/master/docs/source/dedoc_api_usage/return_format.md

This JSON structure shows the 'attachments' field populated when parsing a document with the 'with_attachments' parameter enabled.

```json
  "attachments": [
    {
      "content": {
        "structure": {
          "node_id": "0",
          "text": "",
          "annotations": [],
          "metadata": {
            "paragraph_type": "root",
            "page_id": 0,
            "line_id": 0
          },
          "subparagraphs": []
        },
        "tables": []
      },
      "metadata": {
        "uid": "attach_7098fafc-e566-46d5-9125-adb6e9b047d8",
        "file_name": "image1.png",
        "temporary_file_name": "1714647118_301.png",
        "size": 14874,
        "modified_time": 1714647118,
        "created_time": 1714647118,
        "access_time": 1714647118,
        "file_type": "image/png"
      },
      "version": "2.2",
      "warnings": [],
      "attachments": []
    }
  ]
```

--------------------------------

### JSON Structure Example (Linear)

Source: https://github.com/ispras/dedoc/blob/master/docs/source/dedoc_api_usage/return_format.md

This JSON structure represents the linear output format for document content, highlighting the hierarchy of nodes.

```json
  "content": {
    "structure": {
      "node_id": "0",
      "text": "",
      "annotations": [],
      "metadata": {
        "paragraph_type": "root",
        "page_id": 0,
        "line_id": 0
      },
      "subparagraphs": [
        {
          "node_id": "0.0",
          "text": "Document example",
          "annotations": [
            {
              "start": 0,
              "end": 16,
              "name": "indentation",
              "value": "0"
            },

```

```json
          "node_id": "0.1",
          "text": "Heading",
```