### Install pdfdeal with uv Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html Create a virtual environment with uv, activate it (Linux or Windows), and install pdfdeal. ```bash uv venv source .venv/bin/activate # For Linux source .venv/Scripts/activate # For Windows uv pip install --upgrade pdfdeal ``` -------------------------------- ### uv pip install --upgrade pdfdeal Source: https://noedgeai.github.io/pdfdeal-docs/guide Installs or upgrades the pdfdeal library using uv. ```APIDOC ## uv pip install --upgrade pdfdeal ### Description Installs or upgrades the pdfdeal library using uv. ### Method INSTALL ### Endpoint uv pip install --upgrade pdfdeal ### Parameters None ### Request Example ``` uv pip install --upgrade pdfdeal ``` ### Response #### Success Response (200) - **status** (string) - Installation successful #### Response Example ``` Successfully installed pdfdeal ``` ``` -------------------------------- ### Install pdfdeal with conda Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html Create a conda environment named 'rag' with Python 3.12, activate it, and install pdfdeal. ```bash conda create -n rag python=3.12 conda activate rag pip install --upgrade pdfdeal ``` -------------------------------- ### pip install --upgrade pdfdeal Source: https://noedgeai.github.io/pdfdeal-docs/guide Installs or upgrades the pdfdeal library using pip. ```APIDOC ## pip install --upgrade pdfdeal ### Description Installs or upgrades the pdfdeal library using pip. ### Method INSTALL ### Endpoint pip install --upgrade pdfdeal ### Parameters None ### Request Example ``` pip install --upgrade pdfdeal ``` ### Response #### Success Response (200) - **status** (string) - Installation successful #### Response Example ``` Successfully installed pdfdeal ``` ``` -------------------------------- ### AliCloud OSS upload example Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html Initialize Ali_OSS with your AliCloud credentials and use it with md_replace_imgs to replace images in a markdown file. Requires the oss2 package and public-readable OSS permissions. ```python from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS from pdfdeal.file_tools import md_replace_imgs ossupload = Ali_OSS( OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"), OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"), Endpoint=os.environ.get("Endpoint"), Bucket=os.environ.get("Bucket"), ) md_replace_imgs( mdfile="Output/1706.03762-2024-08-11 17-06-35.md", replace=ossupload, threads=5, ) # Or you want to replace the images of all MD documents in a specified path with the OSS address. # mds_replace_imgs( # path="Output", # replace=ossupload, # threads=5, # ) ``` -------------------------------- ### S3 upload example Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html Initialize the S3 uploader with your S3 credentials and use it with md_replace_imgs to replace images in a markdown file. Requires boto3 and public-readable S3 permissions. ```python from pdfdeal.FileTools.Img.S3 import S3 from pdfdeal.file_tools import md_replace_imgs ossupload = S3( S3_ACCESS_KEY_ID=os.environ.get("S3_ACCESS_KEY_ID"), S3_ACCESS_KEY_SECRET=os.environ.get("S3_ACCESS_KEY_SECRET"), Endpoint=os.environ.get("S3_Endpoint"), Bucket=os.environ.get("S3_Bucket"), Customized_Domain=os.environ.get("S3_Customized_Domain"), ) md_replace_imgs( mdfile="Output/1706.03762-2024-08-11 17-06-35.md", replace=ossupload, threads=5, ) # Or you want to replace the images of all MD documents in a specified path with S3 addresses. # mds_replace_imgs( # path="Output", # replace=ossupload, # threads=5, # ) ``` -------------------------------- ### gen_folder_list example Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Gen_folder.html Generates a list of PDF files in the specified folder recursively and prints the result. ```python files = gen_folder_list("/path/to/folder", "pdf", True) print(files) ``` -------------------------------- ### pip install --upgrade pdfdeal (conda) Source: https://noedgeai.github.io/pdfdeal-docs/guide Installs or upgrades the pdfdeal library using pip within the conda environment. ```APIDOC ## pip install --upgrade pdfdeal (conda) ### Description Installs or upgrades the pdfdeal library using pip within the conda environment. ### Method INSTALL ### Endpoint pip install --upgrade pdfdeal ### Parameters None ### Request Example ``` pip install --upgrade pdfdeal ``` ### Response #### Success Response (200) - **status** (string) - Installation successful #### Response Example ``` Successfully installed pdfdeal ``` ``` -------------------------------- ### Change output_format from 'latex' to 'tex' in pdf2file Source: https://noedgeai.github.io/pdfdeal-docs/changes/v1tov2.html In version 0.4.X, the output_format parameter for LaTeX export must be changed from 'latex' to 'tex'. The example shows the same call in both versions. ```python from pdfdeal import Doc2X client = Doc2X() filepath, _, _ = client.pdf2file( "tests/pdf/sample.pdf", output_format="latex" ) print(filepath) ``` ```python from pdfdeal import Doc2X client = Doc2X() filepath, _, _ = client.pdf2file( "tests/pdf/sample.pdf", output_format="tex" ) print(filepath) ``` -------------------------------- ### Simplify folder processing in pdf2file Source: https://noedgeai.github.io/pdfdeal-docs/changes/v1tov2.html In version 0.4.X, pdf2file automatically recognizes folder paths and maintains the original file structure, eliminating the need for get_files and manual file list handling. The example shows the old and new approaches. ```python from pdfdeal import Doc2X from pdfdeal import get_files Client = Doc2X() file_list, rename_list = get_files( path="./tests/pdf", mode="pdf", out="docx" ) success, failed, flag = Client.pdf2file( pdf_file=file_list, output_path="./Output/newfolder", output_names=rename_list, output_format="docx", ) print(success) print(failed) print(flag) ``` ```python from pdfdeal import Doc2X Client = Doc2X() success, failed, flag = Client.pdf2file( pdf_file="./tests/pdf", output_path="./Output/newfolder", output_format="docx", ) print(success) print(failed) print(flag) ``` -------------------------------- ### Ali_OSS Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html Initializes the Ali_OSS upload tool with AliCloud credentials and uses it with md_replace_imgs to replace images in markdown files. ```APIDOC ## Ali_OSS ### Description Initializes the Ali_OSS upload tool for AliCloud Object Storage Service. Requires the `oss2` package and public read permissions on the bucket. ### Method Class initialization ### Endpoint N/A ### Parameters #### Initialization Parameters - **OSS_ACCESS_KEY_ID** (str) - Required - Your AliCloud ACCESS_KEY ID. - **OSS_ACCESS_KEY_SECRET** (str) - Required - Your AliCloud ACCESS_KEY SECRET. - **Endpoint** (str) - Required - Your OSS Service Endpoint. - **Bucket** (str) - Required - Your OSS Bucket name. ### Example ```python from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS from pdfdeal.file_tools import md_replace_imgs ossupload = Ali_OSS( OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"), OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"), Endpoint=os.environ.get("Endpoint"), Bucket=os.environ.get("Bucket"), ) md_replace_imgs( mdfile="Output/1706.03762-2024-08-11 17-06-35.md", replace=ossupload, threads=5, ) # Or you want to replace the images of all MD documents in a specified path with the OSS address. # mds_replace_imgs( # path="Output", # replace=ossupload, # threads=5, # ) ``` ``` -------------------------------- ### Enable INFO logging for httpx and root logger Source: https://noedgeai.github.io/pdfdeal-docs/changes Set the httpx logger to WARNING and configure the root logger to INFO to see processing output. Place this after importing logging and before running PDFdeal operations. ```python import logging httpx_logger = logging.getLogger("httpx") httpx_logger.setLevel(logging.WARNING) logging.basicConfig(level=logging.INFO) ``` -------------------------------- ### source .venv/bin/activate # For Linux Source: https://noedgeai.github.io/pdfdeal-docs/guide Activates the virtual environment on Linux systems. ```APIDOC ## source .venv/bin/activate # For Linux ### Description Activates the virtual environment on Linux systems. ### Method INSTALL ### Endpoint source .venv/bin/activate # For Linux ### Parameters None ### Request Example ``` source .venv/bin/activate # For Linux ``` ### Response #### Success Response (200) - **status** (string) - Virtual environment activated successfully #### Response Example ``` Successfully activated virtual environment ``` ``` -------------------------------- ### from pdfdeal.Doc2X.Convert import * Source: https://noedgeai.github.io/pdfdeal-docs/guide Imports the encapsulated asynchronous request functions from the Doc2X module. ```APIDOC ## from pdfdeal.Doc2X.Convert import * ### Description Imports the encapsulated asynchronous request functions from the Doc2X module. ### Method IMPORT ### Endpoint from pdfdeal.Doc2X.Convert import * ### Parameters None ### Request Example ``` from pdfdeal.Doc2X.Convert import * ``` ### Response #### Success Response (200) - **functions** (list) - List of imported functions #### Response Example ``` Imported functions: ... ``` ``` -------------------------------- ### source .venv/Scripts/activate # For Windows Source: https://noedgeai.github.io/pdfdeal-docs/guide Activates the virtual environment on Windows systems. ```APIDOC ## source .venv/Scripts/activate # For Windows ### Description Activates the virtual environment on Windows systems. ### Method INSTALL ### Endpoint source .venv/Scripts/activate # For Windows ### Parameters None ### Request Example ``` source .venv/Scripts/activate # For Windows ``` ### Response #### Success Response (200) - **status** (string) - Virtual environment activated successfully #### Response Example ``` Successfully activated virtual environment ``` ``` -------------------------------- ### Complete PDF to Markdown Processing Program Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html This code snippet shows the complete program for converting PDF files to Markdown using the pdfdeal library, including file retrieval, conversion, unzipping, splitting, and image replacement with Alibaba Cloud OSS. ```APIDOC ## Complete Program ### Description This program demonstrates the full workflow for converting PDF files to Markdown using the pdfdeal library, including file retrieval, conversion, unzipping, splitting, and image replacement with Alibaba Cloud OSS. ### Code Example ```python from pdfdeal import Doc2X from pdfdeal.file_tools import get_files, unzips, auto_split_mds, mds_replace_imgs from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS import os Client = Doc2X() out_type = "md" file_list, rename_list = get_files(path="./Files", mode="pdf", out=out_type) success, failed, flag = Client.pdf2file( pdf_file=file_list, output_path="./Output", output_names=rename_list, output_format=out_type, ) print(success, failed, flag) zips = [] for file in success: if file.endswith(".zip"): zips.append(file) success, failed, flag = unzips(zip_paths=zips) print(success, failed, flag) succese, failed, flag = auto_split_mds(mdpath="./Output", out_type="replace") print(succese, failed, flag) ossupload = Ali_OSS( OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"), OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"), Endpoint=os.environ.get("Endpoint"), Bucket=os.environ.get("Bucket"), ) succese, failed, flag = mds_replace_imgs( path="Output", replace=ossupload, threads=5, ) print(succese, failed, flag) ``` ``` -------------------------------- ### S3 Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html Initializes the S3 upload tool with S3 credentials and uses it with md_replace_imgs to replace images in markdown files. ```APIDOC ## S3 ### Description Initializes the S3 upload tool for S3-compatible object storage. Requires the `boto3` package and public read permissions on the bucket. ### Method Class initialization ### Endpoint N/A ### Parameters #### Initialization Parameters - **S3_ACCESS_KEY_ID** (str) - Required - Your S3 ACCESS_KEY ID. - **S3_ACCESS_KEY_SECRET** (str) - Required - Your S3 ACCESS_KEY SECRET. - **Endpoint** (str) - Required - Your S3 service endpoint. - **Bucket** (str) - Required - Your S3 Bucket name. - **Customized_Domain** (str) - Required - Your S3 customized domain name. Note that `{Customized_Domain}/{remote_file_path}` will be returned as the final image address. Please don't forget to add `http://` or `https://` prefix. ### Example ```python from pdfdeal.FileTools.Img.S3 import S3 from pdfdeal.file_tools import md_replace_imgs ossupload = S3( S3_ACCESS_KEY_ID=os.environ.get("S3_ACCESS_KEY_ID"), S3_ACCESS_KEY_SECRET=os.environ.get("S3_ACCESS_KEY_SECRET"), Endpoint=os.environ.get("S3_Endpoint"), Bucket=os.environ.get("S3_Bucket"), Customized_Domain=os.environ.get("S3_Customized_Domain"), ) md_replace_imgs( mdfile="Output/1706.03762-2024-08-11 17-06-35.md", replace=ossupload, threads=5, ) # Or you want to replace the images of all MD documents in a specified path with S3 addresses. # mds_replace_imgs( # path="Output", # replace=ossupload, # threads=5, # ) ``` ``` -------------------------------- ### python -m doc2x -h Source: https://noedgeai.github.io/pdfdeal-docs/guide Displays help information for the doc2x command-line tool. ```APIDOC ## python -m doc2x -h ### Description Displays help information for the doc2x command-line tool. ### Method GET ### Endpoint python -m doc2x -h ### Parameters None ### Request Example ``` python -m doc2x -h ``` ### Response #### Success Response (200) - **help_text** (string) - Help information for the doc2x command #### Response Example ``` Usage: doc2x [OPTIONS] ... ``` ``` -------------------------------- ### S3 upload import Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html Import the S3 class from pdfdeal.FileTools.Img.S3. This is the first step to use S3 upload functionality. ```python from pdfdeal.FileTools.Img.S3 import S3 ``` -------------------------------- ### uv venv Source: https://noedgeai.github.io/pdfdeal-docs/guide Creates a new virtual environment using uv. ```APIDOC ## uv venv ### Description Creates a new virtual environment using uv. ### Method INSTALL ### Endpoint uv venv ### Parameters None ### Request Example ``` uv venv ``` ### Response #### Success Response (200) - **status** (string) - Virtual environment created successfully #### Response Example ``` Successfully created virtual environment ``` ``` -------------------------------- ### conda create -n pdfdeal python=3.11 Source: https://noedgeai.github.io/pdfdeal-docs/guide Creates a new conda environment named 'pdfdeal' with Python 3.11. ```APIDOC ## conda create -n pdfdeal python=3.11 ### Description Creates a new conda environment named 'pdfdeal' with Python 3.11. ### Method INSTALL ### Endpoint conda create -n pdfdeal python=3.11 ### Parameters None ### Request Example ``` conda create -n pdfdeal python=3.11 ``` ### Response #### Success Response (200) - **status** (string) - Environment created successfully #### Response Example ``` Successfully created environment 'pdfdeal' ``` ``` -------------------------------- ### conda activate pdfdeal Source: https://noedgeai.github.io/pdfdeal-docs/guide Activates the conda environment named 'pdfdeal'. ```APIDOC ## conda activate pdfdeal ### Description Activates the conda environment named 'pdfdeal'. ### Method INSTALL ### Endpoint conda activate pdfdeal ### Parameters None ### Request Example ``` conda activate pdfdeal ``` ### Response #### Success Response (200) - **status** (string) - Environment activated successfully #### Response Example ``` Successfully activated environment 'pdfdeal' ``` ``` -------------------------------- ### Upload images to Ali OSS and replace Markdown paths Source: https://noedgeai.github.io/pdfdeal-docs/demo/RAG_pre.html Uploads images referenced in Markdown files to Ali OSS and replaces local paths with URLs. Requires the `oss2` package and pre-configured environment variables for OSS credentials. The `mds_replace_imgs` function processes files in the 'Output' directory with 5 threads and returns success, failure, and flag lists. ```python from pdfdeal.FileTools.Img.Ali_OSS import Ali_OSS from pdfdeal.file_tools import mds_replace_imgs import os ossupload = Ali_OSS( OSS_ACCESS_KEY_ID=os.environ.get("OSS_ACCESS_KEY_ID"), OSS_ACCESS_KEY_SECRET=os.environ.get("OSS_ACCESS_KEY_SECRET"), Endpoint=os.environ.get("Endpoint"), Bucket=os.environ.get("Bucket"), ) succese, failed, flag = mds_replace_imgs( path="Output", replace=ossupload, threads=5, ) print(succese, failed, flag) ``` -------------------------------- ### md_replace_imgs Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/MD_imgs.html Replaces image links in a single Markdown file. Downloads online images to local or uses a custom replacement function. ```APIDOC ## md_replace_imgs ### Description Replaces image links in a single Markdown file (CDN Links -> Local Files/AliOSS/Custom). ### Parameters - **mdfile** (str) - Required - Markdown file path - **replace** (str or function) - Default: "local" - Strings or **functions** used to replace image links. Only accepts "local" - **skip** (str) - Default: None - URLs that start with this string will be skipped. For example, "https://noedgeai.github.io/pdfdeal-docs" - **outputpath** (str) - Default: "" - Saves the output path of the image. If not set, a folder with the same name as the Markdown file will be created with `_img` added. **Only works if `replace` is "local"** - **relative** (bool) - Default: False - Saves images using relative paths. **Valid only if `replace` is "local"** - **threads** (int) - Default: 5 - Number of threads to download the image ### Return Value - **bool** - Returns `True` if all images were downloaded successfully, otherwise returns `False` ### Notes - The `outputpath` and `relative` parameters are valid when `replace` is "local". - If `outputpath` is not set, a folder with the same name as the Markdown file and the addition of `_img` is automatically created to hold the images. ### Example ```python from pdfdeal.file_tools import md_replace_imgs md_replace_imgs( mdfile="Output/1706.03762-2024-08-11 17-06-35.md", outputpath="./ABC" replace="local", threads=5, ) ``` ``` -------------------------------- ### Custom upload function Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html Defines a custom upload function that accepts local_file_path and remote_file_path, and returns a tuple with the accessible URL and a boolean indicating success. ```APIDOC ## Custom Upload Function ### Description Define a custom function to upload files. The function must accept `local_file_path` and `remote_file_path` parameters and return a tuple containing the accessible URL and a boolean indicating success. ### Method Function definition ### Endpoint N/A ### Parameters #### Function Parameters - **local_file_path** (str) - Required - The path of the local file to upload. - **remote_file_path** (str) - Required - The path of the remote file to upload to. ### Return Value - **tuple** - A tuple containing the URL of the uploaded file (str) and a boolean indicating whether the upload was successful (bool). ### Example ```python def upload_file(local_file_path, remote_file_path): """Upload a file Args: local_file_path (str): The path of the local file to upload. remote_file_path (str): The path of the remote file to upload to. Returns: tuple: A tuple containing the URL of the uploaded file and a boolean indicating whether the upload was successful. """ return ("This is a test", True) ``` ``` -------------------------------- ### mds_replace_imgs Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/MD_imgs.html Replaces image links in all Markdown files in a specified directory. Downloads online images to local or uses a custom replacement function. ```APIDOC ## mds_replace_imgs ### Description Replaces image links in all Markdown files in the specified path (CDN Links -> Local Files/AliOSS/Custom). ### Parameters - **path** (str) - Required - Markdown file path - **replace** (str or function) - Default: "local" - Strings or **functions** used to replace image links. Only accepts "local" - **outputpath** (str) - Default: "" - Save the output path of the image. If not set, a folder with the same name as the Markdown file will be created with `_img` added. **Only works if `replace` is "local"** - **relative** (bool) - Default: False - Whether to save the image as a relative path. **Only valid if `replace` is "local"** - **skip** (str) - Default: None - URLs starting with this string will be skipped. For example, "https://noedgeai.github.io/pdfdeal-docs" - **threads** (int) - Default: 2 - Number of MD documents processed simultaneously - **down_load_threads** (int) - Default: 3 - Number of threads downloading images in a Markdown file ### Return Value Returns a tuple `(list1, list2, bool)` with three elements: 1. **list1** (list) - A list of successfully processed Markdown file paths. - element is the path to the processed file (string) 2. **list2** (list) - A list of files that failed to be processed. - element is a dictionary with two keys: - `'error'`: error message (string) - `'path'`: path to the file that failed processing (string) - The value of both keys is the empty string if processing was successful. 3. **bool** (bool) - Processing state - `True`: All files were processed successfully. - `False`: At least one file was not processed. ### Notes - `list1` and `list2` are the same length. - The `outputpath` and `relative` parameters are only valid if `replace` is "local". ### Example ```python mds_replace_imgs( path="Output", replace="local", skip="https://noedgeai.github.io/pdfdeal-docs", threads=5, ) ``` ``` -------------------------------- ### Custom upload function definition Source: https://noedgeai.github.io/pdfdeal-docs/guide/Tools/Upload.html Define a function that accepts local_file_path and remote_file_path, and returns a tuple of the accessible URL and a boolean indicating success. This function is passed to the Document Image Processing Tool in replace. ```python def upload_file(local_file_path, remote_file_path): """Upload a file Args: local_file_path (str): The path of the local file to upload. remote_file_path (str): The path of the remote file to upload to. Returns: tuple: A tuple containing the URL of the uploaded file and a boolean indicating whether the upload was successful. """ return ("This is a test",True) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.