pdfdeal

https://noedgeai.github.io/pdfdeal-docs
Admin
pdfdeal is a Python tool designed to simplify processing PDFs and images, extracting text via OCR,...
Tokens:10,886
Snippets:89
Trust Score:9.7
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install pdfdeal with uv

Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html

Create a virtual environment with uv, activate it (Linux or Windows), and install pdfdeal.

```bash
uv venv
source .venv/bin/activate # For Linux
source .venv/Scripts/activate # For Windows
uv pip install --upgrade pdfdeal
```

--------------------------------

### uv pip install --upgrade pdfdeal

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Installs or upgrades the pdfdeal library using uv.

```APIDOC
## uv pip install --upgrade pdfdeal

### Description
Installs or upgrades the pdfdeal library using uv.

### Method
INSTALL

### Endpoint
uv pip install --upgrade pdfdeal

### Parameters
None

### Request Example
```
uv pip install --upgrade pdfdeal
```

### Response
#### Success Response (200)
- **status** (string) - Installation successful

#### Response Example
```
Successfully installed pdfdeal
```
```

--------------------------------

### Install pdfdeal with conda

Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html

Create a conda environment named 'rag' with Python 3.12, activate it, and install pdfdeal.

```bash
conda create -n rag python=3.12
conda activate rag
pip install --upgrade pdfdeal
```

--------------------------------

### pip install --upgrade pdfdeal

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Installs or upgrades the pdfdeal library using pip.

```APIDOC
## pip install --upgrade pdfdeal

### Description
Installs or upgrades the pdfdeal library using pip.

### Method
INSTALL

### Endpoint
pip install --upgrade pdfdeal

### Parameters
None

### Request Example
```
pip install --upgrade pdfdeal
```

### Response
#### Success Response (200)
- **status** (string) - Installation successful

#### Response Example
```
Successfully installed pdfdeal
```
```

--------------------------------

### AliCloud OSS upload example

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html

Initialize Ali_OSS with your AliCloud credentials and use it with md_replace_imgs to replace images in a markdown file. Requires the oss2 package and public-readable OSS permissions.

```python
from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS
from pdfdeal.file_tools import md_replace_imgs

ossupload = Ali_OSS(
    OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"),
    OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"),
    Endpoint=os.environ.get("Endpoint"),
    Bucket=os.environ.get("Bucket"),
)

md_replace_imgs(
    mdfile="Output/1706.03762-2024-08-11 17-06-35.md",
    replace=ossupload,
    threads=5,
)

# Or you want to replace the images of all MD documents in a specified path with the OSS address.
# mds_replace_imgs(
#     path="Output",
#     replace=ossupload,
#     threads=5,
# )
```

--------------------------------

### S3 upload example

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html

Initialize the S3 uploader with your S3 credentials and use it with md_replace_imgs to replace images in a markdown file. Requires boto3 and public-readable S3 permissions.

```python
from pdfdeal.FileTools.Img.S3 import S3
from pdfdeal.file_tools import md_replace_imgs

ossupload = S3(
    S3_ACCESS_KEY_ID=os.environ.get("S3_ACCESS_KEY_ID"),
    S3_ACCESS_KEY_SECRET=os.environ.get("S3_ACCESS_KEY_SECRET"),
    Endpoint=os.environ.get("S3_Endpoint"),
    Bucket=os.environ.get("S3_Bucket"),
    Customized_Domain=os.environ.get("S3_Customized_Domain"),
)

md_replace_imgs(
    mdfile="Output/1706.03762-2024-08-11 17-06-35.md",
    replace=ossupload,
    threads=5,
)

# Or you want to replace the images of all MD documents in a specified path with S3 addresses.
# mds_replace_imgs(
#     path="Output",
#     replace=ossupload,
#     threads=5,
# )
```

--------------------------------

### gen_folder_list example

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Gen_folder.html

Generates a list of PDF files in the specified folder recursively and prints the result.

```python
files = gen_folder_list("/path/to/folder", "pdf", True)
print(files)
```

--------------------------------

### pip install --upgrade pdfdeal (conda)

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Installs or upgrades the pdfdeal library using pip within the conda environment.

```APIDOC
## pip install --upgrade pdfdeal (conda)

### Description
Installs or upgrades the pdfdeal library using pip within the conda environment.

### Method
INSTALL

### Endpoint
pip install --upgrade pdfdeal

### Parameters
None

### Request Example
```
pip install --upgrade pdfdeal
```

### Response
#### Success Response (200)
- **status** (string) - Installation successful

#### Response Example
```
Successfully installed pdfdeal
```
```

--------------------------------

### Change output_format from 'latex' to 'tex' in pdf2file

Source: https://noedgeai.github.io/pdfdeal-docs/changes/v1tov2.html

In version 0.4.X, the output_format parameter for LaTeX export must be changed from 'latex' to 'tex'. The example shows the same call in both versions.

```python
from pdfdeal import Doc2X

client = Doc2X()
filepath, _, _ = client.pdf2file(
    "tests/pdf/sample.pdf", output_format="latex"
)
print(filepath)
```

```python
from pdfdeal import Doc2X

client = Doc2X()
filepath, _, _ = client.pdf2file(
    "tests/pdf/sample.pdf", output_format="tex"
)
print(filepath)
```

--------------------------------

### Simplify folder processing in pdf2file

Source: https://noedgeai.github.io/pdfdeal-docs/changes/v1tov2.html

In version 0.4.X, pdf2file automatically recognizes folder paths and maintains the original file structure, eliminating the need for get_files and manual file list handling. The example shows the old and new approaches.

```python
from pdfdeal import Doc2X
from pdfdeal import get_files
Client = Doc2X()
file_list, rename_list = get_files(
    path="./tests/pdf", mode="pdf", out="docx"
)
success, failed, flag = Client.pdf2file(
    pdf_file=file_list,
    output_path="./Output/newfolder",
    output_names=rename_list,
    output_format="docx",
)
print(success)
print(failed)
print(flag)
```

```python
from pdfdeal import Doc2X

Client = Doc2X()
success, failed, flag = Client.pdf2file(
    pdf_file="./tests/pdf",
    output_path="./Output/newfolder",
    output_format="docx",
)
print(success)
print(failed)
print(flag)
```

--------------------------------

### Ali_OSS

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html

Initializes the Ali_OSS upload tool with AliCloud credentials and uses it with md_replace_imgs to replace images in markdown files.

```APIDOC
## Ali_OSS

### Description
Initializes the Ali_OSS upload tool for AliCloud Object Storage Service. Requires the `oss2` package and public read permissions on the bucket.

### Method
Class initialization

### Endpoint
N/A

### Parameters
#### Initialization Parameters
- **OSS_ACCESS_KEY_ID** (str) - Required - Your AliCloud ACCESS_KEY ID.
- **OSS_ACCESS_KEY_SECRET** (str) - Required - Your AliCloud ACCESS_KEY SECRET.
- **Endpoint** (str) - Required - Your OSS Service Endpoint.
- **Bucket** (str) - Required - Your OSS Bucket name.

### Example
```python
from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS
from pdfdeal.file_tools import md_replace_imgs

ossupload = Ali_OSS(
    OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"),
    OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"),
    Endpoint=os.environ.get("Endpoint"),
    Bucket=os.environ.get("Bucket"),
)

md_replace_imgs(
    mdfile="Output/1706.03762-2024-08-11 17-06-35.md",
    replace=ossupload,
    threads=5,
)

# Or you want to replace the images of all MD documents in a specified path with the OSS address.
# mds_replace_imgs(
#     path="Output",
#     replace=ossupload,
#     threads=5,
# )
```
```

--------------------------------

### Enable INFO logging for httpx and root logger

Source: https://noedgeai.github.io/pdfdeal-docs/changes

Set the httpx logger to WARNING and configure the root logger to INFO to see processing output. Place this after importing logging and before running PDFdeal operations.

```python
import logging
httpx_logger = logging.getLogger("httpx")
httpx_logger.setLevel(logging.WARNING)
logging.basicConfig(level=logging.INFO)
```

--------------------------------

### source .venv/bin/activate # For Linux

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Activates the virtual environment on Linux systems.

```APIDOC
## source .venv/bin/activate # For Linux

### Description
Activates the virtual environment on Linux systems.

### Method
INSTALL

### Endpoint
source .venv/bin/activate # For Linux

### Parameters
None

### Request Example
```
source .venv/bin/activate # For Linux
```

### Response
#### Success Response (200)
- **status** (string) - Virtual environment activated successfully

#### Response Example
```
Successfully activated virtual environment
```
```

--------------------------------

### from pdfdeal.Doc2X.Convert import *

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Imports the encapsulated asynchronous request functions from the Doc2X module.

```APIDOC
## from pdfdeal.Doc2X.Convert import *

### Description
Imports the encapsulated asynchronous request functions from the Doc2X module.

### Method
IMPORT

### Endpoint
from pdfdeal.Doc2X.Convert import *

### Parameters
None

### Request Example
```
from pdfdeal.Doc2X.Convert import *
```

### Response
#### Success Response (200)
- **functions** (list) - List of imported functions

#### Response Example
```
Imported functions: ...
```
```

--------------------------------

### source .venv/Scripts/activate # For Windows

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Activates the virtual environment on Windows systems.

```APIDOC
## source .venv/Scripts/activate # For Windows

### Description
Activates the virtual environment on Windows systems.

### Method
INSTALL

### Endpoint
source .venv/Scripts/activate # For Windows

### Parameters
None

### Request Example
```
source .venv/Scripts/activate # For Windows
```

### Response
#### Success Response (200)
- **status** (string) - Virtual environment activated successfully

#### Response Example
```
Successfully activated virtual environment
```
```

--------------------------------

### Complete PDF to Markdown Processing Program

Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html

This code snippet shows the complete program for converting PDF files to Markdown using the pdfdeal library, including file retrieval, conversion, unzipping, splitting, and image replacement with Alibaba Cloud OSS.

```APIDOC
## Complete Program

### Description
This program demonstrates the full workflow for converting PDF files to Markdown using the pdfdeal library, including file retrieval, conversion, unzipping, splitting, and image replacement with Alibaba Cloud OSS.

### Code Example
```python
from pdfdeal import Doc2X
from pdfdeal.file_tools import get_files, unzips, auto_split_mds, mds_replace_imgs
from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS
import os

Client = Doc2X()
out_type = "md"
file_list, rename_list = get_files(path="./Files", mode="pdf", out=out_type)
success, failed, flag = Client.pdf2file(
    pdf_file=file_list,
    output_path="./Output",
    output_names=rename_list,
    output_format=out_type,
)
print(success, failed, flag)

zips = []
for file in success:
    if file.endswith(".zip"):
        zips.append(file)
success, failed, flag = unzips(zip_paths=zips)
print(success, failed, flag)

succese, failed, flag = auto_split_mds(mdpath="./Output", out_type="replace")
print(succese, failed, flag)

ossupload = Ali_OSS(
    OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"),
    OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"),
    Endpoint=os.environ.get("Endpoint"),
    Bucket=os.environ.get("Bucket"),
)

succese, failed, flag = mds_replace_imgs(
    path="Output",
    replace=ossupload,
    threads=5,
)
print(succese, failed, flag)
```
```

--------------------------------

### S3

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html

Initializes the S3 upload tool with S3 credentials and uses it with md_replace_imgs to replace images in markdown files.

```APIDOC
## S3

### Description
Initializes the S3 upload tool for S3-compatible object storage. Requires the `boto3` package and public read permissions on the bucket.

### Method
Class initialization

### Endpoint
N/A

### Parameters
#### Initialization Parameters
- **S3_ACCESS_KEY_ID** (str) - Required - Your S3 ACCESS_KEY ID.
- **S3_ACCESS_KEY_SECRET** (str) - Required - Your S3 ACCESS_KEY SECRET.
- **Endpoint** (str) - Required - Your S3 service endpoint.
- **Bucket** (str) - Required - Your S3 Bucket name.
- **Customized_Domain** (str) - Required - Your S3 customized domain name. Note that `{Customized_Domain}/{remote_file_path}` will be returned as the final image address. Please don't forget to add `http://` or `https://` prefix.

### Example
```python
from pdfdeal.FileTools.Img.S3 import S3
from pdfdeal.file_tools import md_replace_imgs

ossupload = S3(
    S3_ACCESS_KEY_ID=os.environ.get("S3_ACCESS_KEY_ID"),
    S3_ACCESS_KEY_SECRET=os.environ.get("S3_ACCESS_KEY_SECRET"),
    Endpoint=os.environ.get("S3_Endpoint"),
    Bucket=os.environ.get("S3_Bucket"),
    Customized_Domain=os.environ.get("S3_Customized_Domain"),
)

md_replace_imgs(
    mdfile="Output/1706.03762-2024-08-11 17-06-35.md",
    replace=ossupload,
    threads=5,
)

# Or you want to replace the images of all MD documents in a specified path with S3 addresses.
# mds_replace_imgs(
#     path="Output",
#     replace=ossupload,
#     threads=5,
# )
```
```

--------------------------------

### python -m doc2x -h

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Displays help information for the doc2x command-line tool.

```APIDOC
## python -m doc2x -h

### Description
Displays help information for the doc2x command-line tool.

### Method
GET

### Endpoint
python -m doc2x -h

### Parameters
None

### Request Example
```
python -m doc2x -h
```

### Response
#### Success Response (200)
- **help_text** (string) - Help information for the doc2x command

#### Response Example
```
Usage: doc2x [OPTIONS] ...
```
```

--------------------------------

### S3 upload import

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html

Import the S3 class from pdfdeal.FileTools.Img.S3. This is the first step to use S3 upload functionality.

```python
from pdfdeal.FileTools.Img.S3 import S3
```

--------------------------------

### uv venv

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Creates a new virtual environment using uv.

```APIDOC
## uv venv

### Description
Creates a new virtual environment using uv.

### Method
INSTALL

### Endpoint
uv venv

### Parameters
None

### Request Example
```
uv venv
```

### Response
#### Success Response (200)
- **status** (string) - Virtual environment created successfully

#### Response Example
```
Successfully created virtual environment
```
```

--------------------------------

### conda create -n pdfdeal python=3.11

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Creates a new conda environment named 'pdfdeal' with Python 3.11.

```APIDOC
## conda create -n pdfdeal python=3.11

### Description
Creates a new conda environment named 'pdfdeal' with Python 3.11.

### Method
INSTALL

### Endpoint
conda create -n pdfdeal python=3.11

### Parameters
None

### Request Example
```
conda create -n pdfdeal python=3.11
```

### Response
#### Success Response (200)
- **status** (string) - Environment created successfully

#### Response Example
```
Successfully created environment 'pdfdeal'
```
```

--------------------------------

### conda activate pdfdeal

Source: https://noedgeai.github.io/pdfdeal-docs/guide

Activates the conda environment named 'pdfdeal'.

```APIDOC
## conda activate pdfdeal

### Description
Activates the conda environment named 'pdfdeal'.

### Method
INSTALL

### Endpoint
conda activate pdfdeal

### Parameters
None

### Request Example
```
conda activate pdfdeal
```

### Response
#### Success Response (200)
- **status** (string) - Environment activated successfully

#### Response Example
```
Successfully activated environment 'pdfdeal'
```
```

--------------------------------

### Upload images to Ali OSS and replace Markdown paths

Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html

Uploads images referenced in Markdown files to Ali OSS and replaces local paths with URLs. Requires the `oss2` package and pre-configured environment variables for OSS credentials. The `mds_replace_imgs` function processes files in the 'Output' directory with 5 threads and returns success, failure, and flag lists.

```python
from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS
from pdfdeal.file_tools import mds_replace_imgs
import os

ossupload = Ali_OSS(
    OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"),
    OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"),
    Endpoint=os.environ.get("Endpoint"),
    Bucket=os.environ.get("Bucket"),
)

succese, failed, flag = mds_replace_imgs(
    path="Output",
    replace=ossupload,
    threads=5,
)
print(succese, failed, flag)
```

--------------------------------

### md_replace_imgs

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/MD_imgs.html

Replaces image links in a single Markdown file. Downloads online images to local or uses a custom replacement function.

```APIDOC
## md_replace_imgs

### Description
Replaces image links in a single Markdown file (CDN Links -> Local Files/AliOSS/Custom).

### Parameters
- **mdfile** (str) - Required - Markdown file path
- **replace** (str or function) - Default: "local" - Strings or **functions** used to replace image links. Only accepts "local"
- **skip** (str) - Default: None - URLs that start with this string will be skipped. For example, "https://noedgeai.github.io/pdfdeal-docs"
- **outputpath** (str) - Default: "" - Saves the output path of the image. If not set, a folder with the same name as the Markdown file will be created with `_img` added. **Only works if `replace` is "local"**
- **relative** (bool) - Default: False - Saves images using relative paths. **Valid only if `replace` is "local"**
- **threads** (int) - Default: 5 - Number of threads to download the image

### Return Value
- **bool** - Returns `True` if all images were downloaded successfully, otherwise returns `False`

### Notes
- The `outputpath` and `relative` parameters are valid when `replace` is "local".
- If `outputpath` is not set, a folder with the same name as the Markdown file and the addition of `_img` is automatically created to hold the images.

### Example
```python
from pdfdeal.file_tools import md_replace_imgs
md_replace_imgs(
    mdfile="Output/1706.03762-2024-08-11 17-06-35.md",
    outputpath="./ABC"
    replace="local",
    threads=5,
)
```
```

--------------------------------

### Custom upload function

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html

Defines a custom upload function that accepts local_file_path and remote_file_path, and returns a tuple with the accessible URL and a boolean indicating success.

```APIDOC
## Custom Upload Function

### Description
Define a custom function to upload files. The function must accept `local_file_path` and `remote_file_path` parameters and return a tuple containing the accessible URL and a boolean indicating success.

### Method
Function definition

### Endpoint
N/A

### Parameters
#### Function Parameters
- **local_file_path** (str) - Required - The path of the local file to upload.
- **remote_file_path** (str) - Required - The path of the remote file to upload to.

### Return Value
- **tuple** - A tuple containing the URL of the uploaded file (str) and a boolean indicating whether the upload was successful (bool).

### Example
```python
def upload_file(local_file_path, remote_file_path):
    """Upload a file

    Args:
        local_file_path (str): The path of the local file to upload.
        remote_file_path (str): The path of the remote file to upload to.

    Returns:
        tuple: A tuple containing the URL of the uploaded file and a boolean indicating whether the upload was successful.
    """
    return ("This is a test", True)
```
```

--------------------------------

### mds_replace_imgs

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/MD_imgs.html

Replaces image links in all Markdown files in a specified directory. Downloads online images to local or uses a custom replacement function.

```APIDOC
## mds_replace_imgs

### Description
Replaces image links in all Markdown files in the specified path (CDN Links -> Local Files/AliOSS/Custom).

### Parameters
- **path** (str) - Required - Markdown file path
- **replace** (str or function) - Default: "local" - Strings or **functions** used to replace image links. Only accepts "local"
- **outputpath** (str) - Default: "" - Save the output path of the image. If not set, a folder with the same name as the Markdown file will be created with `_img` added. **Only works if `replace` is "local"**
- **relative** (bool) - Default: False - Whether to save the image as a relative path. **Only valid if `replace` is "local"**
- **skip** (str) - Default: None - URLs starting with this string will be skipped. For example, "https://noedgeai.github.io/pdfdeal-docs"
- **threads** (int) - Default: 2 - Number of MD documents processed simultaneously
- **down_load_threads** (int) - Default: 3 - Number of threads downloading images in a Markdown file

### Return Value
Returns a tuple `(list1, list2, bool)` with three elements:
1. **list1** (list) - A list of successfully processed Markdown file paths.
   - element is the path to the processed file (string)
2. **list2** (list) - A list of files that failed to be processed.
   - element is a dictionary with two keys:
     - `'error'`: error message (string)
     - `'path'`: path to the file that failed processing (string)
   - The value of both keys is the empty string if processing was successful.
3. **bool** (bool) - Processing state
   - `True`: All files were processed successfully.
   - `False`: At least one file was not processed.

### Notes
- `list1` and `list2` are the same length.
- The `outputpath` and `relative` parameters are only valid if `replace` is "local".

### Example
```python
mds_replace_imgs(
    path="Output",
    replace="local",
    skip="https://noedgeai.github.io/pdfdeal-docs",
    threads=5,
)
```
```

--------------------------------

### Custom upload function definition

Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html

Define a function that accepts local_file_path and remote_file_path, and returns a tuple of the accessible URL and a boolean indicating success. This function is passed to the Document Image Processing Tool in replace.

```python
def upload_file(local_file_path, remote_file_path):
    """Upload a file

    Args:
        local_file_path (str): The path of the local file to upload.
        remote_file_path (str): The path of the remote file to upload to.

    Returns:
        tuple: A tuple containing the URL of the uploaded file and a boolean indicating whether the upload was successful.
    """
    return ("This is a test",True)
```
Tokens:
Raw