Unstructured

https://docs.unstructured.io/
Admin
Unstructured is a platform and toolkit for ingesting and processing unstructured documents to enable...
Tokens:978,932
Snippets:10,268
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Unstructured Python SDK

Source: https://docs.unstructured.io/llms-full.txt

Install the Unstructured client library using pip.

```APIDOC
## Installation

Before using the SDK to interact with Unstructured, install the library:

```bash
pip install unstructured-client
```
```

--------------------------------

### Install Libraries with uv

Source: https://docs.unstructured.io/examplecode/tools/pii

Installs boto3, presidio_analyzer, and presidio_anonymizer using the uv package manager.

```bash
uv add boto3 presidio_analyzer presidio_anonymizer
```

--------------------------------

### Install Unstructured JavaScript/TypeScript SDK

Source: https://docs.unstructured.io/llms-full.txt

Install the unstructured-client library using npm. It is advised to pin your installed version due to semantic versioning.

```bash
npm install unstructured-client
```

--------------------------------

### List Jobs with curl

Source: https://docs.unstructured.io/llms-full.txt

Examples of listing jobs using curl. Shows how to make a basic GET request to the /jobs endpoint and how to filter by workflow ID or job status using query parameters.

```bash
curl --request 'GET' --location \
    "$UNSTRUCTURED_API_URL/jobs" \
    --header 'accept: application/json' \
    --header "unstructured-api-key: $UNSTRUCTURED_API_KEY"
```

```bash
curl --request 'GET' --location \
    "$UNSTRUCTURED_API_URL/jobs?workflow_id=<workflow-id>" \
    --header 'accept: application/json' \
    --header "unstructured-api-key: $UNSTRUCTURED_API_KEY"
```

```bash
curl --request 'GET' --location \
    "$UNSTRUCTURED_API_URL/job?status=<status>" \
    --header 'accept: application/json' \
    --header "unstructured-api-key: $UNSTRUCTURED_API_KEY"
```

--------------------------------

### Install uv using install script

Source: https://docs.unstructured.io/examplecode/tools/mcp-partition

Install the 'uv' package manager using the provided installation script. Remember to restart your terminal after execution. This method is available for macOS, Linux, and Windows.

```bash
# For macOS and Linux:
curl -LsSf https://astral.sh/uv/install.sh | sh
```

```powershell
# For Windows:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```

--------------------------------

### Basic Partitioning Example

Source: https://docs.unstructured.io/llms-full.txt

Example demonstrating how to use the Unstructured Python SDK to send a PDF document to the Partition Endpoint for processing.

```APIDOC
## Basics

Let's start with a simple example in which you send a PDF document to the Unstructured Partition Endpoint to be partitioned by Unstructured.

```python
import os, json

import unstructured_client
from unstructured_client.models import operations, shared

client = unstructured_client.UnstructuredClient(
    api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
)

filename = "PATH_TO_INPUT_FILE"

req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=shared.Files(
            content=open(filename, "rb"),
            file_name=filename,
        ),
        strategy=shared.Strategy.VLM,
        vlm_model="gpt-4o",
        vlm_model_provider="openai",
        languages=['eng'],
        split_pdf_page=True,            # If True, splits the PDF file into smaller chunks of pages.
        split_pdf_allow_failed=True,    # If True, the partitioning continues even if some pages fail.
        split_pdf_concurrency_level=15  # Set the number of concurrent request to the maximum value: 15.
    ),
)

try:
    res = client.general.partition(
        request=req
    )
    element_dicts = [element for element in res.elements]
    
    # Print the processed data's first element only.
    print(element_dicts[0])

    # Write the processed data to a local file.
    json_elements = json.dumps(element_dicts, indent=2)

    with open("PATH_TO_OUTPUT_FILE", "w") as file:
        file.write(json_elements)
except Exception as e:
    print(e)
```
```

--------------------------------

### Install Unstructured with All Document Support

Source: https://docs.unstructured.io/open-source/installation/full-installation

Use this command to install the Unstructured library with support for all file types. This is recommended for comprehensive functionality.

```bash
uv add "unstructured[all-docs]"
```

--------------------------------

### Install MCP Server Dependencies

Source: https://docs.unstructured.io/examplecode/tools/crewai

Navigate to the cloned MCP server directory and install its dependencies using uv.

```bash
cd UNS-MCP
uv sync
```

--------------------------------

### Install Notion Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `notion` extra to enable the Notion source connector.

```bash
pip install "unstructured-ingest[notion]"
```

--------------------------------

### Install uv with wget

Source: https://docs.unstructured.io/examplecode/tools/crewai

Installs the uv package manager on macOS and Linux using wget.

```bash
wget -qO- https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Install PPTX Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `pptx` extra to enable support for processing .pptx files.

```bash
pip install "unstructured-ingest[pptx]"
```

--------------------------------

### Install Project Dependencies

Source: https://docs.unstructured.io/examplecode/tools/mcp

With the virtual environment activated, install the project's code dependencies using the 'uv sync' command.

```bash
uv sync
```

--------------------------------

### Install JIRA Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `jira` extra to enable the JIRA source connector.

```bash
pip install "unstructured-ingest[jira]"
```

--------------------------------

### Install PPT Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `ppt` extra to enable support for processing .ppt files.

```bash
pip install "unstructured-ingest[ppt]"
```

--------------------------------

### Install SharePoint Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `sharepoint` extra to enable the SharePoint source connector.

```bash
pip install "unstructured-ingest[sharepoint]"
```

--------------------------------

### Install GitLab Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `gitlab` extra to enable the GitLab source connector.

```bash
pip install "unstructured-ingest[gitlab]"
```

--------------------------------

### Partition Endpoint Examples

Source: https://docs.unstructured.io/llms.txt

Examples of accessing the Unstructured Partition Endpoint via different methods.

```APIDOC
## Partition Endpoint Examples

### Description
This page provides some examples of accessing Unstructured Partition Endpoint via different methods.

### Usage
Refer to the linked documentation for specific examples using various tools and languages.
```

--------------------------------

### Install Google Drive Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `google-drive` extra to enable the Google Drive source connector.

```bash
pip install "unstructured-ingest[google-drive]"
```

--------------------------------

### Python Weaviate Client Example

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/weaviate

Example demonstrating the use of the weaviate-client Python package to connect to and interact with a Weaviate database.

```python
import weaviate

client = weaviate.Client(
    url=os.getenv("WEAVIATE_URL"),
    auth_client_secret=weaviate.AuthApiKey(
        api_key=os.getenv("WEAVIATE_API_KEY")
    ),
    additional_headers={
        "inference-addr": os.getenv("WEAVIATE_INFERENCE_ADDR")
    },
)

```

--------------------------------

### Install GitHub Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `github` extra to enable the GitHub source connector.

```bash
pip install "unstructured-ingest[github]"
```

--------------------------------

### Install EPUB Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `epub` extra to enable support for processing EPUB files.

```bash
pip install "unstructured-ingest[epub]"
```

--------------------------------

### Install Salesforce Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `salesforce` extra to enable the Salesforce source connector.

```bash
pip install "unstructured-ingest[salesforce]"
```

--------------------------------

### Install DOC Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `doc` extra to enable support for processing .doc files.

```bash
pip install "unstructured-ingest[doc]"
```

--------------------------------

### Install Confluence Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `confluence` extra to enable the Confluence source connector.

```bash
pip install "unstructured-ingest[confluence]"
```

--------------------------------

### Install PDF Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `pdf` extra to enable support for processing PDF files.

```bash
pip install "unstructured-ingest[pdf]"
```

--------------------------------

### Install OneDrive Connector

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/onedrive

Install the OneDrive connector by running this command. Additional dependencies may be required based on your specific needs.

```bash
pip install "unstructured-ingest[onedrive]"
```

--------------------------------

### Install DOCX Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `docx` extra to enable support for processing .docx files.

```bash
pip install "unstructured-ingest[docx]"
```

--------------------------------

### Install Unstructured Python SDK

Source: https://docs.unstructured.io/api-reference/workflow

Install the Unstructured Python SDK. Ensure you are within your Python virtual environment.

```bash
pip install "unstructured-client>=0.30.6"
```

--------------------------------

### Ingest Data to Local Weaviate via CLI

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/weaviate

Example CLI command to ingest data from a local directory to a locally installed Weaviate instance. Includes optional chunking and embedding configurations.

```bash
#!/usr/bin/env bash

# Chunking and embedding is optional.

# For Weaviate installed locally:
unstructured-ingest \
  local \
    --input-path $LOCAL_FILE_INPUT_DIR \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --chunking-strategy by_title \
    --embedding-provider huggingface \
    --strategy hi_res \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  weaviate-local \
    --collection $WEAVIATE_COLLECTION
```

--------------------------------

### Create and Navigate to Project Directory

Source: https://docs.unstructured.io/api-reference/quickstart

Create a new directory for your project and change into it. This sets up the workspace for the quickstart.

```bash
mkdir unstructured_api_quickstart
cd unstructured_api_quickstart
```

--------------------------------

### Install Astra DB Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `astra` extra to enable the Astra DB source connector.

```bash
pip install "unstructured-ingest[astra]"
```

--------------------------------

### Initialize and Navigate Project Directory

Source: https://docs.unstructured.io/examplecode/tools/mcp

Use `uv init` to create a new project directory and `cd` to enter it. This sets up the basic project structure.

```bash
uv init mcp-unstructured-demo
cd mcp-unstructured-demo
```

--------------------------------

### MongoDB Connection String Example

Source: https://docs.unstructured.io/api-reference/workflow/destinations/mongodb

Example of a MongoDB Atlas connection string. Ensure all placeholders are filled before use.

```text
mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
```

--------------------------------

### Create Qdrant Collection with Python Client

Source: https://docs.unstructured.io/llms-full.txt

Use the Python Qdrant Client to create a collection on a Qdrant Cloud cluster. This example configures the collection for vectors with 3072 dimensions. Ensure you have the qdrant-client library installed.

```python
from qdrant_client import QdrantClient, models
import os

client = QdrantClient(

```

--------------------------------

### Initialize Unstructured Client and Prepare File for Partitioning

Source: https://docs.unstructured.io/llms-full.txt

Sets up the Unstructured Python client using API key and URL from environment variables. It then reads a local file in binary mode to prepare it for sending to the Unstructured API for partitioning.

```python
from unstructured_client import UnstructuredClient
from unstructured_client.models import operations, shared
import os

input_filepath = "local-ingest-input-pdf/Portable-Document-Format-Specifications.pdf"

client = UnstructuredClient(
    api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
)

with open(input_filepath, "rb") as f:
    files = shared.Files(
        content=f.read(),
        file_name=input_filepath
    )

# Chunk the document with a basic chunking strategy.
# Create chunks that are at least around 200 characters (soft limit)

```

--------------------------------

### Example JSON Metadata from Unstructured

Source: https://docs.unstructured.io/llms-full.txt

Illustrates the structure of metadata produced by Unstructured, including extraction status, coordinates, file type, and data source information. This JSON can guide the creation of corresponding database schema fields.

```json
{
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
}
```

--------------------------------

### Run Unstructured Partition Endpoint Quickstart

Source: https://docs.unstructured.io/llms-full.txt

Process local files using the Unstructured Partition Endpoint. Configure your API key, input directory or files, and output directory. The code will create the output directory if it doesn't exist.

```python
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models import shared

# Replace with your Unstructured API key
api_key = os.environ.get("<unstructured-api-key>")

# Initialize the Unstructured client
client = UnstructuredClient(api_key_auth=api_key)

# Define input and output paths
# To process all files within a directory, change None to a string that contains the path to the directory on your local machine.
input_dir = None

# To process specific files within a directory or across multiple directories, change None to a string that contains a comma-separated list of filepaths on your local machine.
input_file = None

# Specify a string that contains the path to the directory on your local machine that you want Unstructured to send its JSON output files.
output_dir = "./output"

# Create the output directory if it does not exist
os.makedirs(output_dir, exist_ok=True)

# Prepare the files to be processed
files = shared.Files(
    content=None, 
    file_name=None
)

# Define the request to the Unstructured API
request = shared.PartitionRequest(
    files=files,
    # Set input_dir to process all files in a directory
    # input_dir=input_dir,
    # Set input_file to process specific files
    # input_file=input_file,
    # Specify the output directory
    output_dir=output_dir,
    # You can also specify other parameters like strategy, chunking_strategy, etc.
    # strategy="fast",
    # chunking_strategy="basic",
)

# Send the request to the Unstructured API
response = client.general.partition(request=request)

# Process the response
if response.status_code == 200:
    print("File processed successfully!")
    # You can access the processed data from response.elements
    # print(response.elements)
else:
    print(f"Error processing file: {response.status_code}")
    print(response.text)

```

--------------------------------

### Initialize Project Directory

Source: https://docs.unstructured.io/examplecode/tools/mcp-partition

Creates and navigates into a new project directory for the MCP partition demo.

```bash
uv init mcp-unstructured-partition-demo
cd mcp-unstructured-partition-demo
```

--------------------------------

### Ingest to LanceDB Google Cloud Storage

Source: https://docs.unstructured.io/llms-full.txt

This command is for ingesting data into LanceDB OSS using Google Cloud Storage. It requires the GCS service account key and the bucket URI. The example shows basic partitioning and embedding setup.

```bash
# For LanceDB OSS with data storage in a Google Cloud Storage bucket:
unstructured-ingest \
  local \
    --input-path $LOCAL_FILE_INPUT_DIR \
    --chunking-strategy by_title \
    --embedding-provider huggingface \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-gcs \
      --google-service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE
```

--------------------------------

### Create and Navigate Project Directory

Source: https://docs.unstructured.io/examplecode/tools/crewai

Use these commands to create a new project directory and move into it.

```bash
mkdir crewai_unstructured_demo
cd crewai_unstructured_demo
```

--------------------------------

### Query PDF with HuggingChat (JavaScript/TypeScript)

Source: https://docs.unstructured.io/examplecode/codesamples/api/huggingchat

This TypeScript example demonstrates partitioning a PDF and querying HuggingChat. It includes setup for the Unstructured client and Hugging Face login, and a helper function to stream responses. Note: Unstructured recommends the Python Ingest library for better performance.

```typescript
import { UnstructuredClient } from "unstructured-client";
import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
import { PartitionResponse } from "unstructured-client/sdk/models/operations";

import * as fs from "fs";
import * as path from 'path';
import { fileURLToPath } from 'url';
import { Login, ChatBot} from "huggingface-chat";

const key = process.env.UNSTRUCTURED_API_KEY || '';
const url = process.env.UNSTRUCTURED_API_URL || '';
const username = process.env.HUGGING_FACE_EMAIL || '';
const userpassword = process.env.HUGGING_FACE_PASSWORD || '';
const cookieDirPath = process.env.HUGGING_FACE_COOKIE_DIR_PATH || '';

const inputFilepath = path.dirname(fileURLToPath(import.meta.url)) +
                          "/local-ingest-pdf-source/constitution.pdf"

async function getChatBot(email: string, password: string, cachePath: string): Promise<ChatBot> {
    const signin = new Login(email, password)
    const result = await signin.login(cachePath)
    const chat = new ChatBot(result)

    return chat
}

async function queryAndResponse(chatbot: ChatBot, query: string) {
    let data = await chatbot.chat(query)

    if (! data) {
        console.log("No data available.");
        return;
    }
    
    let reader = data.stream?.getReader();

    if (! reader) {
        console.log("No stream available.");
        return;
    }

    while (true) {
        const { done, value } = await reader?.read();
        if (done) break;
        process.stdout.write(value);
    }    
}

const client = new UnstructuredClient({
    security: { apiKeyAuth: key },
    serverURL: new URL(url).origin
});

const data = fs.readFileSync(inputFilepath);

client.general.partition({
    partitionParameters: {
        files: {
            content: data,
            fileName: inputFilepath
        },
        strategy: Strategy.HiRes,
        splitPdfPage: true,
        splitPdfAllowFailed: true,
        splitPdfConcurrencyLevel: 15
    }
}).then(async (res: PartitionResponse) => {
    if (res.statusCode == 200) {
        let voting_texts = ""

        if (res)
            for (const element of res) {
                if (element.text.includes("vot")) {
                    voting_texts += " " + element.text;
                }
        }

        let chat = await getChatBot(username, userpassword, cachePath)
        
        await chat.initialize()

        const firstQuery = 'Given the following information, what is the minimum voting age in the United States?'
        const secondQuery = 'And when were women given the right to vote in the United States?'

        console.log(`${firstQuery} ${voting_texts}\n\n-----\n`)
        await queryAndResponse(chat, firstQuery)
        console.log(`\n\n-----\n`)
        console.log(`${secondQuery}\n\n-----\n`)
        await queryAndResponse(chat, secondQuery)
    }    
}).catch((e) => {
    if (e.statusCode) {
        console.log(e.statusCode);
        console.log(e.body);
    } else {
        console.log(e);
    }
});
```
```

--------------------------------

### Partition DOCX file using partition

Source: https://docs.unstructured.io/llms-full.txt

Demonstrates creating a DOCX file and then partitioning it using the `partition` function with both filename and file-like object inputs.

```python
import docx

from unstructured.partition.auto import partition

document = docx.Document()
document.add_paragraph("Important Analysis", style="Heading 1")
document.add_paragraph("Here is my first thought.", style="Body Text")
document.add_paragraph("Here is my second thought.", style="Normal")
document.save("mydoc.docx")

elements = partition(filename="mydoc.docx")

with open("mydoc.docx", "rb") as f:
    elements = partition(file=f)

```

--------------------------------

### Install Zendesk Connector

Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/zendesk

Install the Zendesk connector and its dependencies using pip. Additional dependencies may be required based on specific needs.

```bash
pip install "unstructured-ingest[zendesk]"
```

--------------------------------

### Install AWS SDK for Python with uv

Source: https://docs.unstructured.io/examplecode/tools/firecrawl

Install the boto3 package using uv. This SDK is used for interacting with AWS services, such as S3.

```bash
uv add boto3
```

--------------------------------

### Install LanceDB Ingestion Dependencies

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/lancedb

Install the necessary dependencies for using the LanceDB connector with unstructured-ingest. This command installs the core package and the LanceDB-specific extras.

```bash
pip install "unstructured-ingest[lancedb]"
```

--------------------------------

### Project Directory Structure

Source: https://docs.unstructured.io/api-reference/quickstart

This shows the expected project directory structure for the quickstart.

```text
unstructured_api_quickstart/
  ├── .venv/
  ├── input/  <- Upload your input files here.
  ├── output/ <- Unstructured will download its output files here.
  ├── .gitignore
  ├── .python-version
  ├── main.py <- Your Python code will go here.
  ├── pyproject.toml
  ├── README.md
  └── uv.lock
```

--------------------------------

### Get Workflow Channel

Source: https://docs.unstructured.io/llms.txt

Get workflow-scoped notification channel by ID.

```APIDOC
## Get Workflow Channel

### Description
Get workflow-scoped notification channel by ID.

### Method
GET

### Endpoint
/workflows/{workflow_id}/channels/{channel_id}

### Parameters
#### Path Parameters
- **workflow_id** (string) - Required - The ID of the workflow.
- **channel_id** (string) - Required - The ID of the notification channel to retrieve.
```

--------------------------------

### Create and Get Qdrant Collection

Source: https://docs.unstructured.io/api-reference/workflow/destinations/qdrant

Demonstrates how to initialize a Qdrant client, create a new collection with specified vector parameters, and then retrieve the collection's status. Ensure Qdrant URL and API key are set as environment variables.

```python
from qdrant_client import QdrantClient, models
import os

client = QdrantClient(
    url=os.getenv("QDRANT_URL"),
    api_key=os.getenv("QDRANT_API_KEY")
)

client.create_collection(
    collection_name=os.getenv("QDRANT_COLLECTION"),
    vectors_config=models.VectorParams(
        size=3072,
        distance=models.Distance.COSINE
    )
)

collection = client.get_collection(
                 collection_name=os.getenv("QDRANT_COLLECTION")
             )

print(f"The collection named '{os.getenv("QDRANT_COLLECTION")}' exists and " +
      f"has a status of '{collection.status}'.")
```

--------------------------------

### Install Vectara Connector

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/vectara

Install the Vectara connector and its dependencies using pip.

```bash
pip install "unstructured-ingest[vectara]"
```

--------------------------------

### Create and Navigate to Project Directory

Source: https://docs.unstructured.io/examplecode/tools/firecrawl

Create a new directory for your project and change into it. This organizes your project files.

```bash
mkdir firecrawl_unstructured_demo
cd firecrawl_unstructured_demo
```

--------------------------------

### Install Dependencies

Source: https://docs.unstructured.io/examplecode/tools/neo4j-chatbot

Installs the necessary Python packages for Unstructured.io client, Neo4j, and neo4j-graphrag.

```python
!pip install unstructured-client
!pip install neo4j
!pip install neo4j-graphrag

```

--------------------------------

### POST Request Example

Source: https://docs.unstructured.io/llms-full.txt

This example demonstrates how to change the partition strategy to 'vlm' and specify an OpenAI model (gpt-4o) using a cURL command.

```APIDOC
## POST / 

### Description
This endpoint allows you to partition documents, with options to specify the strategy and the VLM model provider and name.

### Method
POST

### Endpoint
/

### Parameters
#### Request Body
- **files** (file) - Required - The file to be processed.
- **strategy** (string) - Optional - The partitioning strategy to use. Example: `vlm`.
- **vlm_model_provider** (string) - Optional - The provider of the VLM model. Example: `openai`.
- **vlm_model** (string) - Optional - The name of the VLM model to use. Example: `gpt-4o`.

### Request Example
```bash
curl -X 'POST' $UNSTRUCTURED_API_URL \
 -H 'accept: application/json' \
 -H 'Content-Type: multipart/form-data' \
 -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
 -F 'files=@sample-docs/layout-parser-paper.pdf' \
 -F 'strategy=vlm' \
 -F 'vlm_model_provider=openai' \
 -F 'vlm_model=gpt-4o'
```
```

--------------------------------

### Install Ingest with Weaviate Dependency

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Use this command to install the `unstructured-ingest` package with support for Weaviate.

```bash
pip install "unstructured-ingest[weaviate]"
```

--------------------------------

### Python SDK Example

Source: https://docs.unstructured.io/llms-full.txt

This Python SDK example shows how to partition files asynchronously, specifying the 'vlm' strategy and 'gpt-4o' model, and saving the results to a JSON file.

```APIDOC
## partition_async

### Description
Asynchronously partitions a file using specified parameters, including strategy, VLM model, and PDF splitting options. The results are saved as a JSON file.

### Method
`client.general.partition_async(request)`

### Parameters
#### Request Body (`req`)
- **partition_parameters** (object) - Required - Contains parameters for partitioning.
  - **files** (object) - Required - File details.
    - **content** (binary) - Required - The file content.
    - **file_name** (string) - Required - The name of the file.
  - **strategy** (shared.Strategy) - Required - The partitioning strategy. Example: `shared.Strategy.VLM`.
  - **vlm_model** (string) - Optional - The VLM model to use. Example: `gpt-4o`.
  - **vlm_model_provider** (string) - Optional - The provider of the VLM model. Example: `openai`.
  - **split_pdf_page** (boolean) - Optional - Whether to split PDF pages.
  - **split_pdf_allow_failed** (boolean) - Optional - Whether to allow failed splits.
  - **split_pdf_concurrency_level** (integer) - Optional - Concurrency level for PDF splitting.

### Response
#### Success Response
- **elements** (list) - A list of elements extracted from the document.
```

--------------------------------

### Install Ingest with Slack Dependency

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Use this command to install the `unstructured-ingest` package with support for Slack.

```bash
pip install "unstructured-ingest[slack]"
```

--------------------------------

### Install Snowflake Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `snowflake` extra to enable the Snowflake source connector.

```bash
pip install "unstructured-ingest[snowflake]"
```

--------------------------------

### Run workflow with local source and local/remote destination (Python SDK)

Source: https://docs.unstructured.io/api-reference/workflow/overview

This example demonstrates running a workflow with local input files. Replace `<path/to/input/file>` with the actual path to your file. For local destinations, you'll need to download processed files separately.

```python
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import RunWorkflowRequest
from unstructured_client.models.shared import InputFiles

input_files = []

for filename in [
    "<path/to/input/file>",
    "<path/to/input/file>"
]:
    with open(filename, "rb") as f:
        input_files.append(
            InputFiles(
                content=f.read(),
                file_name=filename
            )
        )

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.workflows.run_workflow(
        request={
            "workflow_id": "<workflow-id>"
            "body_run_workflow": {
                "input_files": input_files
            }
        }
    )

    print(response.raw_response)
```

--------------------------------

### General Partitioning with File-like Object and Page Breaks

Source: https://docs.unstructured.io/llms-full.txt

Shows how to partition a document using a file-like object opened in binary mode. This example also enables the `include_page_breaks` option and displays elements 5 through 15.

```APIDOC
## General Partitioning with File-like Object and Page Breaks

### Description
Partition a document using a file-like object and enable `include_page_breaks` to capture page breaks as `PageBreak` elements. This example displays elements 5 through 15.

### Method
`partition`

### Parameters
#### Path Parameters
None

#### Query Parameters
* `file` (file-like object) - Required - The file object to be partitioned.
* `include_page_breaks` (bool) - Optional - If True, include `PageBreak` elements.

#### Request Body
None

### Request Example
```python
from unstructured.partition.auto import partition
import os

filename = os.path.join(EXAMPLE_DOCS_DIRECTORY, "layout-parser-paper-fast.pdf")
with open(filename, "rb") as f:
  elements = partition(file=f, include_page_breaks=True)
print("\n\n".join([str(el) for el in elements][5:15]))
```

### Response
#### Success Response (200)
- `elements` (list) - A list of `Element` objects representing the partitioned content.
```

--------------------------------

### Install Outlook Connector for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `outlook` extra to enable the Outlook source connector.

```bash
pip install "unstructured-ingest[outlook]"
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://docs.unstructured.io/examplecode/tools/mcp-partition

Sets up a Python virtual environment for project dependencies. Includes commands for macOS/Linux and Windows, and how to deactivate.

```bash
# For macOS and Linux:
uv venv
source .venv/bin/activate

# For Windows:
uv venv
.venv\Scripts\activate
```

```bash
deactivate
```

--------------------------------

### Initialize uv Project

Source: https://docs.unstructured.io/api-reference/quickstart

Initialize a new uv project within your current directory. This command sets up the project structure for uv.

```bash
uv init
```

--------------------------------

### Partition DOCX file using partition

Source: https://docs.unstructured.io/open-source/core-functionality/partitioning

Demonstrates partitioning a .docx file by filename and then by file-like object. Ensure the document is saved before partitioning.

```python
import docx

from unstructured.partition.auto import partition

document = docx.Document()
document.add_paragraph("Important Analysis", style="Heading 1")
document.add_paragraph("Here is my first thought.", style="Body Text")
document.add_paragraph("Here is my second thought.", style="Normal")
document.save("mydoc.docx")

elements = partition(filename="mydoc.docx")

with open("mydoc.docx", "rb") as f:
    elements = partition(file=f)
```

--------------------------------

### Using the 'fast' Partitioning Strategy

Source: https://docs.unstructured.io/llms-full.txt

For simple documents like PDFs without images or tables, consider using the 'fast' partitioning strategy. Test this strategy on a few documents before opting for the more resource-intensive 'hi_res' strategy.

```python
# Example usage (conceptual, actual call depends on SDK context)
# resp = s.general.partition(request=req, strategy='fast')
```

--------------------------------

### Install SingleStore Connector

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/singlestore

Install the SingleStore connector and its dependencies using pip. Additional dependencies may be required based on specific use cases.

```bash
pip install "unstructured-ingest[singlestore]"
```

--------------------------------

### Configure Zendesk Source

Source: https://docs.unstructured.io/api-reference/workflow/sources/zendesk

Use this command to create a Zendesk source connector. Replace placeholders with your specific Zendesk details and desired configuration.

```bash
--header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "zendesk",
      "config": {
          "subdomain": "<subdomain>",
          "email": "<email>",
          "api-token": "<api-token>",
          "item-type": "<item-type>",
          "batch-size": <batch-size>
      }
  }'
```

--------------------------------

### Install Unstructured Ingest with DuckDB

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/duckdb

Install the necessary packages for the unstructured-ingest library with DuckDB support.

```bash
pip install "unstructured-ingest[duckdb]"
```

--------------------------------

### Configure Variables

Source: https://docs.unstructured.io/examplecode/tools/neo4j-chatbot

Sets up essential API keys and connection URIs for Unstructured.io and Neo4j.

```python
unstructured_api_key="UNSTRUCTURED_API_KEY"
neo4j_uri = "NEO4J_URI"
neo4j_database = "neo4j" 
neo4j_user = "neo4j" 
neo4j_password = "NEO4J_PASSWORD"
openai_api_key ="OPENAI_API_KEY"

```

--------------------------------

### Install uv with curl

Source: https://docs.unstructured.io/examplecode/tools/crewai

Installs the uv package manager on macOS and Linux using curl.

```bash
curl -fsSL https://get.uv.dev | bash
```

--------------------------------

### Partition Document with Unstructured SDK (v0.11.0+)

Source: https://docs.unstructured.io/llms-full.txt

This example demonstrates how to use the Unstructured SDK (version 0.11.0 and later) to partition a PDF document. It includes setting up the client, reading a file, and calling the partition method with various parameters like strategy and PDF splitting options. It also shows how to handle the response and potential errors.

```APIDOC
## Partition Document with Unstructured SDK (v0.11.0+)

### Description
This code snippet shows how to partition a document using the Unstructured SDK, version 0.11.0 or newer. It covers initializing the client with API keys and URLs, reading a local file, and sending it to the partition endpoint. Key parameters such as `strategy`, `splitPdfPage`, `splitPdfAllowFailed`, `splitPdfConcurrencyLevel`, and `languages` are demonstrated. The example also includes handling the response and logging any errors.

### Method
`client.general.partition()`

### Parameters
#### Request Body (partitionParameters)
- **files** (object) - Required - The file to be partitioned.
  - **content** (Buffer) - Required - The file content.
  - **fileName** (string) - Required - The name of the file.
- **strategy** (Strategy) - Optional - The partitioning strategy to use (e.g., `Strategy.HiRes`).
- **splitPdfPage** (boolean) - Optional - Whether to split PDF pages for parallel processing. Defaults to `true`.
- **splitPdfAllowFailed** (boolean) - Optional - Whether to continue PDF splitting even if some split operations fail.
- **splitPdfConcurrencyLevel** (number) - Optional - The number of parallel requests for PDF splitting. Max is 15.
- **languages** (string[]) - Optional - A list of languages to use for language detection.

### Request Example
```typescript
import { UnstructuredClient } from "unstructured-client";
import { PartitionResponse } from "unstructured-client/sdk/models/operations";
import { Strategy } from "unstructured-client/sdk/models/shared";
import * as fs from "fs";

const key = process.env.UNSTRUCTURED_API_KEY;
const url = process.env.UNSTRUCTURED_API_URL;

const client = new UnstructuredClient({
    serverURL: url,
    security: {
        apiKeyAuth: key,
    },
});

const filename = "PATH_TO_INPUT_FILE";
const data = fs.readFileSync(filename);

client.general.partition({
    partitionParameters: {
        files: {
            content: data,
            fileName: filename,
        },
        strategy: Strategy.HiRes,
        splitPdfPage: true,
        splitPdfAllowFailed: true,
        splitPdfConcurrencyLevel: 15,
        languages: ['eng']
    }
}).then((res: PartitionResponse) => {
    if (res.statusCode == 200) {
        console.log(res.elements?.[0])
        const jsonElements = JSON.stringify(res, null, 2)
        fs.writeFileSync("PATH_TO_OUTPUT_FILE", jsonElements)
    }
}).catch((e) => {
    if (e.statusCode) {
      console.log(e.statusCode);
      console.log(e.body);
    } else {
      console.log(e);
    }
});
```

### Response
#### Success Response (200)
- **elements** (array) - An array of partitioned elements.

#### Response Example
```json
{
  "elements": [
    {
      "type": "title",
      "element_id": "...",
      "text": "Example Document Title",
      "metadata": {
        "filename": "example.pdf",
        "page_number": 1
      }
    }
  ]
}
```
```

--------------------------------

### Install json-converter Package

Source: https://docs.unstructured.io/llms-full.txt

Install the json-converter package using pip. This is the first step before using its functionalities.

```bash
pip install json-converter
```

--------------------------------

### Install genson package

Source: https://docs.unstructured.io/llms-full.txt

Install the genson package using pip. This is a prerequisite for generating JSON schemas.

```bash
pip install genson
```

--------------------------------

### Install XLSX Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `xlsx` extra to enable support for processing .xlsx files.

```bash
pip install "unstructured-ingest[xlsx]"
```

--------------------------------

### Configure SingleStore Connection and Partitioning

Source: https://docs.unstructured.io/llms-full.txt

Sets up a connection to SingleStore with specific access credentials, host, port, user, and database. It configures partitioning by API, including API key, endpoint, and additional arguments for PDF splitting. Chunking strategy, embedding provider, and local file destination/uploader are also specified.

```python
source_connection_config=SingleStoreConnectionConfig(
              access_config=SingleStoreAccessConfig(
                  password=os.getenv("SINGLESTORE_PASSWORD")
              ),
              host=os.getenv("SINGLESTORE_HOST"),
              port=os.getenv("SINGLESTORE_PORT"),
              user=os.getenv("SINGLESTORE_USER"),
              database=os.getenv("SINGLESTORE_DB")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
```

--------------------------------

### Install TSV Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `tsv` extra to enable support for processing TSV files.

```bash
pip install "unstructured-ingest[tsv]"
```

--------------------------------

### Install RST Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `rst` extra to enable support for processing .rst files.

```bash
pip install "unstructured-ingest[rst]"
```

--------------------------------

### API Processing (CLI)

Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/teradata-sql

Example of using --partition-by-api with --api-key and --partition-endpoint for processing files via the Unstructured API using the CLI.

```bash
--partition-by-api \
  --api-key $UNSTRUCTURED_API_KEY \
  --partition-endpoint $UNSTRUCTURED_API_URL
```

--------------------------------

### Install RTF Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `rtf` extra to enable support for processing .rtf files.

```bash
pip install "unstructured-ingest[rtf]"
```

--------------------------------

### JavaScript/TypeScript SDK Example

Source: https://docs.unstructured.io/llms-full.txt

This TypeScript SDK example demonstrates how to process files from a source path, partition them using the 'HiRes' strategy, and save the output to a destination path.

```APIDOC
## partition

### Description
Partitions files from a given source path, applying the 'HiRes' strategy and PDF splitting options. Processed data is saved to a destination path.

### Method
`client.general.partition(request)`

### Parameters
#### Request Body (`request`)
- **partitionParameters** (object) - Required - Parameters for partitioning.
  - **files** (object) - Required - File details.
    - **content** (Buffer) - Required - The file content.
    - **fileName** (string) - Required - The name of the file.
  - **strategy** (Strategy) - Required - The partitioning strategy. Example: `Strategy.HiRes`.
  - **splitPdfPage** (boolean) - Optional - Whether to split PDF pages.
  - **splitPdfConcurrencyLevel** (number) - Optional - Concurrency level for PDF splitting.
```

--------------------------------

### Create Dropbox Source Connector with Python SDK

Source: https://docs.unstructured.io/llms-full.txt

Example of initializing the Unstructured client and creating a Dropbox source connector using the Python SDK. Ensure your UNSTRUCTURED_API_KEY environment variable is set.

```python
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
from unstructured_client.models.shared import CreateSourceConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",

```

--------------------------------

### Install ODT Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `odt` extra to enable support for processing .odt files.

```bash
pip install "unstructured-ingest[odt]"
```

--------------------------------

### Install Markdown Support for Ingest

Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies

Install the `md` extra to enable support for processing Markdown files.

```bash
pip install "unstructured-ingest[md]"
```

--------------------------------

### Partition Document with Unstructured SDK (SDK <= v0.10.6)

Source: https://docs.unstructured.io/llms-full.txt

This example shows how to partition a document using an older version of the Unstructured SDK (<= v0.10.6). It highlights the differences in parameter handling compared to newer versions, specifically how parameters like `files` and `strategy` were passed directly.

```APIDOC
## Partition Document with Unstructured SDK (SDK <= v0.10.6)

### Description
This code snippet demonstrates partitioning a document using the Unstructured SDK versions prior to v0.11.0. It shows the older method of passing parameters directly to the `partition` method, including the `files` object and the `strategy` as a string.

### Method
`client.general.partition()`

### Parameters
#### Request Body
- **files** (object) - Required - The file to be partitioned.
  - **content** (Buffer) - Required - The file content.
  - **fileName** (string) - Required - The name of the file.
- **strategy** (string) - Optional - The partitioning strategy to use (e.g., `'hi_res'`).
- **languages** (string[]) - Optional - A list of languages to use for language detection.

### Request Example
```typescript
import { UnstructuredClient } from "unstructured-client";
import { PartitionResponse } from "unstructured-client/dist/sdk/models/operations";
import * as fs from "fs";

const key = process.env.UNSTRUCTURED_API_KEY;
const url = process.env.UNSTRUCTURED_API_URL;

const client = new UnstructuredClient({
    serverURL: url,
    security: {
        apiKeyAuth: key,
    },
});

const filename = "PATH_TO_FILE";
const data = fs.readFileSync(filename);

client.general.partition({
    files: {
        content: data,
        fileName: filename,
    },
    strategy: 'hi_res',
    languages: ['eng']
}).then((res: PartitionResponse) => {
    if (res.statusCode == 200) {
        console.log(res.elements?.[0]);
    }
}).catch((e) => {
  if (e.statusCode) {
    console.log(e.statusCode);
    console.log(e.body);
  } else {
    console.log(e);
  }
});
```

### Response
#### Success Response (200)
- **elements** (array) - An array of partitioned elements.

#### Response Example
```json
{
  "elements": [
    {
      "type": "title",
      "element_id": "...",
      "text": "Example Document Title",
      "metadata": {
        "filename": "example.pdf",
        "page_number": 1
      }
    }
  ]
}
```
```

--------------------------------

### Run an on-demand job using cURL

Source: https://docs.unstructured.io/api-reference/workflow/overview

This example shows how to initiate an on-demand job using a cURL command. It details the necessary headers, form data for request parameters, and how to upload one or more files. Remember to replace all placeholder values.

```APIDOC
## Run an on-demand job using cURL

### Description
Initiates an on-demand job via a cURL POST request. This method allows for direct interaction with the API endpoint for job creation and file uploads.

### Method
`POST`

### Endpoint
`/jobs/`

### Parameters
#### Headers
* `accept`: `application/json`
* `unstructured-api-key`: `$UNSTRUCTURED_API_KEY`

#### Form Data
* `request_data` (string) - Required - JSON string containing `template_id` and `job_nodes`.
* `input_files` (file) - Required - The file(s) to be processed. Use the `@` prefix for file paths. Specify `filename` and `type` for each file.

### Request Example
```bash
curl --request 'POST' --location \
'$UNSTRUCTURED_API_URL/jobs/' \
--header 'accept: application/json' \
--header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
--form "request_data=\"{\"template_id\":\"<workflow-template-id>\",\"job_nodes\":[<job-node-settings>]}\"" \
--form "input_files=@</full/path/to/local/filename.extension>;filename=<filename.extension>;type=<local-file-media-type>" \
--form "input_files=@</full/path/to/local/anotherfile.pdf>;filename=<anotherfile.pdf>;type=application/pdf" # For each additional file to be uploaded.
```

### Response
#### Success Response (200)
Returns a JSON object containing `input_file_ids` and `output_node_files` which can be used to download processed file data.

### Note
* Each on-demand job is limited to 10 files, and each file is limited to 10 MB.
* Wait at least one second between launch requests to avoid rate limit errors.
* A maximum of 5 on-demand jobs can run concurrently.
```
Tokens:
Raw