### Install Sycamore AI

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/get_started.rst

Installs the core Sycamore AI library using pip. This is the fundamental step to begin using Sycamore.

```python
pip install sycamore-ai
```

--------------------------------

### Project Setup Instructions

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/remote-processors/README.md

Instructions for setting up the project, including initializing submodules, installing dependencies, generating gRPC/protobuf code, and building Docker images.

```bash
git submodule update --init --recursive
poetry install --no-root
make build_proto
poetry install
docker build -t rps-os -f docker/Dockerfile .
docker build -t rps .
docker compose -f docker/compose.yml up
```

--------------------------------

### Install Sycamore AI with Local Inference

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/get_started.rst

Installs Sycamore AI with the 'local-inference' extra, which is necessary for running inference tasks locally. This allows for offline processing and experimentation.

```python
pip install sycamore-ai[local-inference]
```

--------------------------------

### Getting Started with Storm

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/sycamore/sycamore/tests/resources/data/texts/Ray.txt

This resource provides guidance on getting started with Apache Storm, a distributed real-time computation system.

```APIDOC
Apache Storm:
  Description: A distributed real-time computation system.
  Key Features:
    - Real-time processing
    - Distributed computation
    - Fault tolerance
  Usage: Building and deploying real-time data processing applications.
  References:
    - LEIBIUSKY, J., EISBRUCH, G., AND SIMONASSI, D. Getting Started with Storm. O’Reilly Media, Inc., 2012.
```

--------------------------------

### Project Setup Instructions

Source: https://github.com/aryn-ai/sycamore/blob/main/apps/remote-processor-service/README.md

Instructions for setting up the project, including initializing submodules, installing dependencies, generating gRPC/protobuf code, and building Docker images.

```bash
git submodule update --init --recursive
poetry install --no-root
make build_proto
poetry install
docker build -t rps-os -f docker/Dockerfile .
docker build -t rps .
docker compose -f docker/compose.yml up
```

--------------------------------

### Install Sycamore AI with OpenSearch Connector

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/get_started.rst

Installs Sycamore AI along with the OpenSearch connector, enabling integration with OpenSearch for vector database functionalities. This utilizes pip's extra installation feature.

```python
pip install sycamore-ai[opensearch]
```

--------------------------------

### Example Sycamore CLI Query Result

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/sycamore/sycamore/query/README.md

An example of the output from a Sycamore CLI query, showing the result and a brief explanation.

```bash
Result:

There are 73 incident reports in the database. This number is based on the latest data entry
and reflects the total count of all recorded incidents.
```

--------------------------------

### Create OpenSearch Data Volume

Source: https://github.com/aryn-ai/sycamore/blob/main/apps/opensearch/README.md

Creates a Docker volume named 'opensearch_data' to persist OpenSearch data. This is a one-time setup step.

```bash
docker volume create opensearch_data
```

--------------------------------

### API Key Configuration (Example)

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/minilm-l6-v2_greedy-section-merger_duckdb.ipynb

Provides an example of how to set API keys, recommending storage in configuration files or environment variables for security. This snippet is for quick testing purposes.

```python
# It's best to store API keys in a configuration file or set them as environment variables.
# For quick testing, you can define them here:
#
# os.environ["ARYN_API_KEY"] = "YOUR_ARYN_API_KEY"
```

--------------------------------

### Create OpenSearch Data Volume

Source: https://github.com/aryn-ai/sycamore/blob/main/apps/opensearch/2.11/README.md

Creates a Docker volume named 'opensearch_data' to persist OpenSearch data. This is a one-time setup step.

```bash
docker volume create opensearch_data
```

--------------------------------

### Install Sycamore

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/index.rst

Installs the Sycamore library using pip. This is the basic installation for the core functionality.

```python
pip install sycamore-ai
```

--------------------------------

### Start OpenSearch Docker Compose

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/sycamore/sycamore/query/README.md

Starts the OpenSearch cluster locally using Docker Compose. This is a prerequisite for indexing data with Sycamore.

```bash
cd .. && docker compose up
```

--------------------------------

### Install Sycamore AI with OpenSearch and Local Inference

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/aryn-opensearch-bedrock-rag-example.ipynb

Installs the Sycamore AI library with support for OpenSearch and local inference capabilities. This command ensures all necessary dependencies are met for using Sycamore with these features.

```bash
# Requirements
!pip install "sycamore-ai[opensearch,local-inference]"
```

--------------------------------

### Executing cURL Command with API Key from Environment

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/ArynPartitionerPython.ipynb

Executes a cURL command to partition a PDF file, utilizing the ARYN_API_KEY environment variable for authentication. This example specifies options to disable table structure and image extraction.

```bash
! curl --location 'https://api.aryn.cloud/v1/document/partition' --header "Authorization: Bearer ${ARYN_API_KEY}" --form 'pdf=@"./data/1706.03762.pdf"' --form 'options="{\"extract_table_structure\":false,\"extract_images\":false}"'
```

--------------------------------

### Direct PDF Partitioning with Aryn

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/ArynPartitionerExample.ipynb

This example directly calls the Aryn PDF Partitioner to process a PDF file. It logs the process and prints the partitioned output. The Aryn API key is retrieved using `ArynConfig.get_aryn_api_key()`.

```python
import logging
import os

from sycamore.utils.aryn_config import ArynConfig
from sycamore.transforms.detr_partitioner import ArynPDFPartitioner

logging.getLogger().setLevel(logging.INFO)

doc_path = "../lib/sycamore/sycamore/tests/resources/data/pdfs/Transformer.pdf"

a = ArynPDFPartitioner(model_name_or_path=None)
with open(doc_path, "rb") as file:
    b = a.partition_pdf(file, aryn_api_key=ArynConfig.get_aryn_api_key())
    print(b)
```

--------------------------------

### Install Sycamore with OpenSearch Support

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/text-embedding-3-small_greedy-text-element-merger_opensearch.ipynb

Installs the Sycamore library with the necessary dependencies for OpenSearch integration. This command ensures all required packages are available for using Sycamore's OpenSearch features.

```python
!pip install sycamore-ai[opensearch]
# DocPrep code uses the Sycamore document ETL library: https://github.com/aryn-ai/sycamore
```

--------------------------------

### Visualizing and Saving Document Partitions

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/ArynPartitionerExample.ipynb

This example demonstrates how to partition a PDF from S3, visualize the partitions with bounding boxes, and save the visualized pages as images. It uses `ArynPartitioner` with table structure extraction and `DrawBoxes` to visualize elements. The output images are saved to a local directory.

```python
import sycamore
from sycamore.functions.document import split_and_convert_to_image, DrawBoxes
from sycamore.transforms.partition import ArynPartitioner
from sycamore.utils.image_utils import image_page_filename_fn
from sycamore.utils.pdf_utils import show_pages
from sycamore.utils.aryn_config import ArynConfig
import logging

sycamore.shutdown()  # auto-reload can make this necessary
context = sycamore.init()

# This creates a DocSet and runs the Sycamore Partitioner. You can change the threshold (default is 0.4) or enable OCR.
# You can use this example document: s3://aryn-public/sycamore-partitioner-examples/document-example-1.pdf   #
ds = context.read.binary(
    paths=["s3://aryn-public/sycamore-partitioner-examples/document-example-1.pdf"], binary_format="pdf"
).partition(partitioner=ArynPartitioner(extract_table_structure=True))

# This visualizes partitions inline in the notebook.
show_pages(ds)

os.makedirs("/tmp/example", exist_ok=True)
# To save the visualized partitions for every page, you can use the following transforms.
ds.flat_map(split_and_convert_to_image).map_batch(
    DrawBoxes, f_constructor_kwargs={"draw_table_cells": True}
).write.files("/tmp/example", filename_fn=image_page_filename_fn)
```

--------------------------------

### Install Sycamore

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/sycamore/README.md

Installs the Sycamore Python package. This command is used to get started with the Sycamore library for document processing.

```Shell
pip install sycamore-ai
```

--------------------------------

### Initialize Sycamore and Import Aryn Partitioner

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/VisualizePartitioner.ipynb

Initializes the Sycamore context and imports necessary components for using the Aryn Partitioner. This sets up the environment for document processing.

```python
# This example draws bounding boxes and labels document elements, and it then displays the image as output and writes them to S3.

import sycamore
from sycamore.transforms.partition import ArynPartitioner
from sycamore.utils.pdf_utils import show_pages

context = sycamore.init()
```

--------------------------------

### Environment Setup and Initialization

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/subtask-sample.ipynb

Loads environment variables from a .env file and initializes the Sycamore context. It also sets up an OpenAI LLM instance and a SentenceTransformer embedder.

```python
from dotenv import load_dotenv
import os
import sycamore
from sycamore.evaluation import EvaluationDataPoint
from sycamore.evaluation.pipeline import EvaluationPipeline
from sycamore.transforms.embed import SentenceTransformerEmbedder
from sycamore.transforms.query import OpenSearchQueryExecutor
from sycamore.evaluation.subtasks import SubtaskExecutor
from sycamore.functions import HuggingFaceTokenizer
from sycamore.llms.openai import OpenAI, OpenAIModels
from sycamore.transforms import COALESCE_WHITESPACE
from sycamore.transforms.merge_elements import GreedyTextElementMerger
from sycamore.transforms.partition import ArynPartitioner
from sycamore.connectors.file.materialized_scan import DocScan
from sycamore.docset import DocSet


# add OPENAI_API_KEY to .env
load_dotenv(".env")

path = "amazon_2017_10k.pdf"
index = "sample0"

openai_llm = OpenAI(OpenAIModels.GPT_4O.value)

hf_model = "sentence-transformers/all-mpnet-base-v2"
tokenizer = HuggingFaceTokenizer(hf_model)
embedder = SentenceTransformerEmbedder(model_name=hf_model, batch_size=100)

context = sycamore.init()
```

--------------------------------

### macOS Setup for ttviz Dependencies

Source: https://github.com/aryn-ai/sycamore/blob/main/apps/timetrace/README.md

Installs the 'libgd' dependency using Homebrew on macOS and sets the necessary environment variables (CPATH and LIBRARY_PATH) for the compiler to find the installed library.

```shell
brew install libgd

export CPATH=$HOMEBREW_PREFIX:$CPATH
export LIBRARY_PATH=$HOMEBREW_PREFIX:$LIBRARY_PATH

# Typically HOMEBREW_PREFIX should be set to /opt/homebrew.
```

--------------------------------

### Title Context Template

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/default-prep-script.ipynb

Provides a template for extracting titles from text elements. It includes examples of how to format input text to guide the title extraction process.

```Python
def get_title_context_template():
    # ruff: noqa: E501
    return """
        ELEMENT 1: Jupiter's Moons
        ELEMENT 2: Ganymede 2020
        ELEMENT 3: by Audi Lauper and Serena K. Goldberg. 2011
        ELEMENT 4: From Wikipedia, the free encyclopedia
        ELEMENT 5: Ganymede, or Jupiter III, is the largest and most massive natural satellite of Jupiter as well as in the Solar System, being a planetary-mass moon. It is the largest Solar System object without an atmosphere, despite being the only moon of the Solar System with a magnetic field. Like Titan, it is larger than the planet Mercury, but has somewhat less surface gravity than Mercury, Io or the Moon.
        =========
        "Ganymede 2020"

        ELEMENT 1: FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
        ELEMENT 2: Tarun Kalluri * UCSD
        ELEMENT 3: Deepak Pathak CMU
        ELEMENT 4: Manmohan Chandraker UCSD
        ELEMENT 5: Du Tran Facebook AI
        ELEMENT 6: https://tarun005.github.io/FLAVR/
        ELEMENT 7: 2 2 0 2
        ELEMENT 8: b e F 4 2
        ELEMENT 9: ]
        ELEMENT 10: V C . s c [
        ========
        "FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation"

        """
```

--------------------------------

### Configure Aryn API Key

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/VisualizePartitioner.ipynb

Checks for the Aryn API key, which is required for the Aryn Partitioner service. Provides instructions on how to set the API key via environment variables or a configuration file.

```python
from sycamore.utils.aryn_config import ArynConfig, _DEFAULT_PATH

assert ArynConfig.get_aryn_api_key() != "", f"Unable to find aryn API key.  Looked in {_DEFAULT_PATH}"
```

--------------------------------

### Aryn API Key Setup (YAML)

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/elasticsearch-writer.ipynb

Provides an example of how to configure the Aryn API key in a YAML file. This is a recommended method for securely storing the API key, typically placed at a specified path.

```yaml
aryn_token: "YOUR-ARYN-API-KEY"
```

--------------------------------

### Running the Sycamore Query UI

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/query.rst

Instructions to set up and run the web-based UI for Sycamore Query. This UI allows users to easily experiment with queries, inspect query plans, and debug results. It requires navigating to the UI directory, installing dependencies, and running the main script.

```bash
cd apps/query-ui
poetry install
poetry run queryui/main.py
```

--------------------------------

### LLM Prompt Templates for Title and Author Extraction

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/tutorials/sycamore_jupyter_dev_example.md

Defines prompt templates for a generative AI model to extract titles and authors from document elements. These templates provide examples to guide the AI in identifying and extracting the desired information.

```python
title_context_template = """
    ELEMENT 1: Jupiter's Moons
    ELEMENT 2: Ganymede 2020
    ELEMENT 3: by Audi Lauper and Serena K. Goldberg. 2011
    ELEMENT 4: From Wikipedia, the free encyclopedia
    ELEMENT 5: Ganymede, or Jupiter III, is the largest and most massive natural satellite of Jupiter as well as in the Solar System, being a planetary-mass moon. It is the largest Solar System object without an atmosphere, despite being the only moon of the Solar System with a magnetic field. Like Titan, it is larger than the planet Mercury, but has somewhat less surface gravity than Mercury, Io or the Moon.
    =========
    "Ganymede 2020"

    ELEMENT 1: FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
    ELEMENT 2: Tarun Kalluri * UCSD
    ELEMENT 3: Deepak Pathak CMU
    ELEMENT 4: Manmohan Chandraker UCSD
    ELEMENT 5: Du Tran Facebook AI
    ELEMENT 6: https://tarun005.github.io/FLAVR/
    ELEMENT 7: 2 2 0 2
    ELEMENT 8: b e F 4 2
    ELEMENT 9: ]
    ELEMENT 10: V C . s c [
    ========
    "FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation"

    """
author_context_template = """
    ELEMENT 1: Jupiter's Moons
    ELEMENT 2: Ganymede 2020
    ELEMENT 3: by Audi Lauper and Serena K. Goldberg. 2011
    ELEMENT 4: From Wikipedia, the free encyclopedia
    ELEMENT 5: Ganymede, or Jupiter III, is the largest and most massive natural satellite of Jupiter as well as in the Solar System, being a planetary-mass moon. It is the largest Solar System object without an atmosphere, despite being the only moon of the Solar System with a magnetic field. Like Titan, it is larger than the planet Mercury, but has somewhat less surface gravity than Mercury, Io or the Moon.
    =========
    Audi Laupe, Serena K. Goldberg

    ELEMENT 1: FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
    ELEMENT 2: Tarun Kalluri * UCSD
    ELEMENT 3: Deepak Pathak CMU
    ELEMENT 4: Manmohan Chandraker UCSD
    ELEMENT 5: Du Tran Facebook AI
    ELEMENT 6: https://tarun005.github.io/FLAVR/
    ELEMENT 7: 2 2 0 2
    ELEMENT 8: b e F 4 2
    ELEMENT 9: ]
    ELEMENT 10: V C . s c [
    ========
    Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, Du Tran

  """

```

--------------------------------

### Connect to OpenSearch and Get Info

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/opensearch-writer.ipynb

Initializes the OpenSearch client using provided arguments and prints the cluster information.

```python
# Let's initialize the OpenSearch client and connect it to the database to perform queries
client = OpenSearch(**os_client_args)

print(client.info())
```

--------------------------------

### Sycamore Query Plan Structure

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/query.rst

An example of a Sycamore query plan represented as a JSON object. This plan outlines a sequence of operations, starting with querying a database for specific records and then applying a TopK operation to analyze the results. It details the nodes, their descriptions, inputs, and specific parameters for each operation.

```APIDOC
SycamoreQueryClient.generate_plan() Example:

{
    "nodes": {
        "0": QueryDatabase(
                node_id=0,
                description="Get all the incident reports with substantial aircraft damage",
                input=None,
                index="const_ntsb",
                query={"match": {"properties.entity.aircraftDamage": "Substantial"}}
             ),
        "1": TopK(
                node_id=1,
                description="Get the breakdown of aircraft types",
                input=[0],
                field="properties.entity.aircraft",
                primary_field="properties.entity.accidentNumber",
                K=100,
                descending=False)
    }
}

QueryDatabase:
  node_id: int
  description: str
  input: Optional[List[int]]
  index: str
  query: Dict

TopK:
  node_id: int
  description: str
  input: List[int]
  field: str
  primary_field: str
  K: int
  descending: bool
```

--------------------------------

### Create DocSet and Run Aryn Partitioner

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/VisualizePartitioner.ipynb

Creates a Sycamore DocSet from a PDF file stored in S3 and applies the Aryn Partitioner. Allows configuration of extraction parameters like `extract_table_structure` and `threshold`.

```python
# This creates a DocSet and runs the Aryn Partitioner. You can change the threshold (default is 0.4) or enable OCR.
# You can use this example document: s3://aryn-public/sycamore-partitioner-examples/document-example-1.pdf
ds = context.read.binary(
    paths=["s3://aryn-public/sycamore-partitioner-examples/document-example-1.pdf"], binary_format="pdf"
).partition(partitioner=ArynPartitioner(extract_table_structure=True))
```

--------------------------------

### Install async_timeout for Linux

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/tutorials/sycamore_jupyter_dev_example.md

Installs the async_timeout library, which is required on Linux systems when installing Sycamore data preparation libraries locally to ensure full functionality.

```python
pip install async_timeout
```

--------------------------------

### Sycamore CLI Help

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/sycamore/sycamore/query/README.md

Displays help information and available options for the Sycamore command-line client.

```bash
poetry run ./libs/sycamore/sycamore/query/client.py --help
```

--------------------------------

### Aryn SDK Configuration and API Key Check

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/ArynPartitionerPython.ipynb

Initializes the Aryn SDK configuration, typically by loading API keys from a default path. It includes an assertion to verify that an API key is available, providing guidance on how to set it if it's missing.

```python
from aryn_sdk.client.config import ArynConfig, _DEFAULT_PATH

aryn_config = ArynConfig(aryn_config_path=_DEFAULT_PATH)

assert aryn_config.api_key() != "", f"Unable to find aryn API key.  Looked in {_DEFAULT_PATH}"
os.environ["ARYN_API_KEY"] = aryn_config.api_key() # put in env for curl command below
```

--------------------------------

### Install Missing Packages

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/elasticsearch-writer.ipynb

Installs a specified package using pip. The '-q' flag ensures quiet installation, suppressing most output. This is useful for ensuring all necessary dependencies are available before running the main script.

```bash
!pip install -q PACKAGE_NAME
```

--------------------------------

### Set Aryn API Key (Alternative)

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/VisualizePartitioner.ipynb

Demonstrates an alternative, though less secure, method to set the Aryn API key directly within the notebook environment. This is generally discouraged due to the risk of exposing the key.

```python
import os
os.environ["ARYN_API_KEY"] = "UNSAFE-ARYN-API-KEY-LOCATION"
```

--------------------------------

### Hybrid Search Query Example

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/ndd_example.ipynb

An example of a hybrid search query to OpenSearch, combining keyword matching and neural search. It specifies the fields to retrieve, the query text, generative QA parameters, and the number of results.

```python
query_str = "how does force majeure affect assets and insolvency"
query = {
    "_source": [
        "text_representation",
    ],
    "query": {
        "hybrid": {
            "queries": [
                {
                    "match": {"text_representation": query_str},
                },
                {
                    "neural": {
                        "embedding": {
                            "query_text": query_str,
                            "k": 100,
                            "model_id": get_model_id(),
                        },
                    },
                },
            ],
        },
    },
    "ext": {
        "generative_qa_parameters": {
            "llm_question": query_str,
            "context_size": 10,
            "llm_model": "gpt-4",
        },
    },
    "size": 100,
}
do_query(query)
```

--------------------------------

### Install LangChain

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/tutorials/conversational_memory_with_langchain.md

Installs the LangChain library, which is a prerequisite for using the OpenSearchChatMessageHistory class.

```bash
pip install langchain
```

--------------------------------

### Running OpenSearch and RPS Locally

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/remote-processors/README.md

Instructions for running OpenSearch and the Remote Processor Service locally, including starting OpenSearch in a detached mode and running the RPS service with a specified configuration.

```bash
docker run -d --rm --network=host -e discovery.type=single-node rps-os
poetry run server config/pipelines.yml
```

--------------------------------

### Install Poppler Utilities

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/text-embedding-3-small_greedy-section-merger_pinecone.ipynb

Installs the Poppler utilities package, which is required for PDF processing by Sycamore.

```bash
!apt-get install poppler-utils
```

--------------------------------

### Initialize SycamoreQueryClient

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/query-demo.ipynb

Initializes the SycamoreQueryClient and imports necessary components for querying and visualization. It sets up the console for rich output.

```python
from sycamore.query.client import SycamoreQueryClient
from sycamore.query.visualize import visualize_plan
from rich.console import Console

console = Console()
client = SycamoreQueryClient()
```

--------------------------------

### Install Sycamore with DuckDB Connector

Source: https://github.com/aryn-ai/sycamore/blob/main/README.md

Installs Sycamore with the DuckDB connector for vector database integration.

```bash
pip install sycamore-ai[duckdb]
```

--------------------------------

### Install Git Pre-Commit Hooks

Source: https://github.com/aryn-ai/sycamore/blob/main/CONTRIBUTING.md

Installs the pre-commit framework to automatically run checks (linting, formatting, type-checking, tests) before each commit.

```bash
poetry run pre-commit install

```

--------------------------------

### Setting API Key (Alternative Methods)

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/ArynPartitionerPython.ipynb

Provides alternative methods for setting the Aryn API key if the default configuration fails. This includes setting an environment variable directly or embedding it within the notebook, with a warning about security implications for the latter.

```python
# If the above assertion fails, you can either set the environment variable ARYN_API_KEY and restart jupyter
# or make a yaml file at the specified path in the assertion error that looks like:
# 
# aryn_token: "YOUR-ARYN-API-KEY"
# 
# It is unsafe, but if neither of those options work, you can put it in this notebook with
# 
# import os
# os.environ["ARYN_API_KEY"] = "UNSAFE-ARYN-API-KEY-LOCATION" 
# 
# but beware that it is easy to accidentally commit the notebook file and have it include your key.
```

--------------------------------

### Install Sycamore

Source: https://github.com/aryn-ai/sycamore/blob/main/README.md

Installs the Sycamore Python package. Use extras to include vector database connectors.

```bash
pip install sycamore-ai
```

--------------------------------

### Sycamore Initialization and Imports

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/default-prep-script.ipynb

Imports necessary modules from the sycamore library and initializes an index. This snippet sets up the environment for using Sycamore's partitioning and embedding functionalities.

```Python
from pathlib import Path
import os

import sycamore
from sycamore.functions import HuggingFaceTokenizer, TextOverlapChunker
from sycamore.llms.openai import OpenAI, OpenAIModels
from sycamore.transforms.embed import SentenceTransformerEmbedder
from sycamore.transforms.extract_entity import OpenAIEntityExtractor
from sycamore.transforms.merge_elements import GreedyTextElementMerger
from sycamore.transforms.partition import ArynPartitioner, HtmlPartitioner

index = "demoindex_0"
```

--------------------------------

### Data Preparation with Bash

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/default-prep-script.ipynb

Uses bash commands to create directories and download PDF and HTML files for data preparation. This is a common step before processing data with Sycamore.

```Bash
mkdir -p default-prep-data/pdf default-prep-data/html
curl https://sortbenchmark.org/ELSAR2022.pdf -o default-prep-data/pdf/elsar.pdf
curl https://sortbenchmark.org -o default-prep-data/html/home.html
```

--------------------------------

### Install Matplotlib

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/query-demo.ipynb

Installs the matplotlib library, which is likely used for visualization purposes within the Sycamore query client or related tools.

```python
!pip install matplotlib
```

--------------------------------

### Install Sycamore with OpenSearch Support

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/index.rst

Installs Sycamore along with the necessary dependencies for OpenSearch integration. This allows Sycamore to load data into OpenSearch.

```python
pip install sycamore-ai[opensearch]
```

--------------------------------

### Importing Aryn SDK and Utilities

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/ArynPartitionerPython.ipynb

Imports necessary modules from the aryn-sdk and other standard Python libraries for file operations, subprocess execution, and base64 encoding. These are foundational for interacting with the Aryn API and processing its responses.

```python
import os
import subprocess
import base64

from pathlib import Path
from IPython.display import display

from aryn_sdk.partition import partition_file, table_elem_to_dataframe, draw_with_boxes
```

--------------------------------

### Install Sycamore Dependencies

Source: https://github.com/aryn-ai/sycamore/blob/main/CONTRIBUTING.md

Installs all project dependencies, including extras, using Poetry. It excludes the root project to avoid self-installation issues.

```bash
poetry install --all-extras --no-root

```

--------------------------------

### Testing the Processor via OpenSearch

Source: https://github.com/aryn-ai/sycamore/blob/main/apps/remote-processor-service/README.md

Example of how to test the configured remote processor by sending a search request to OpenSearch with the specified pipeline.

```bash
curl 'http://localhost:9200/demoindex0/_search?search_pipeline=remote_pipeline&pretty' --json ' {
  "query": {
    "match": {
      "text_representation": "armadillo"
    }
  }
}'
```

--------------------------------

### Running OpenSearch and RPS Locally

Source: https://github.com/aryn-ai/sycamore/blob/main/apps/remote-processor-service/README.md

Instructions for running OpenSearch and the Remote Processor Service locally, including starting OpenSearch in a detached mode and running the RPS service with a specified configuration.

```bash
docker run -d --rm --network=host -e discovery.type=single-node rps-os
poetry run server config/pipelines.yml
```

--------------------------------

### Install Sycamore AI with Pinecone Support

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/text-embedding-3-small_greedy-section-merger_pinecone.ipynb

Installs the Sycamore AI library with the necessary dependencies for Pinecone integration using pip.

```bash
!pip install sycamore-ai[pinecone]
```

--------------------------------

### Install Poppler Utilities

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/minilm-l6-v2_greedy-section-merger_duckdb.ipynb

Installs the poppler-utils package, which is often required for PDF processing functionalities within document analysis pipelines.

```bash
!apt-get install poppler-utils
```

--------------------------------

### Install OpenSearch Python Client

Source: https://github.com/aryn-ai/sycamore/blob/main/docs/source/sycamore/tutorials/conversational_memory_with_langchain.md

Installs the necessary Python package for interacting with OpenSearch, which is required for Sycamore's conversation memory.

```bash
pip install opensearch-py
```

--------------------------------

### Testing the Processor via OpenSearch

Source: https://github.com/aryn-ai/sycamore/blob/main/lib/remote-processors/README.md

Example of how to test the configured remote processor by sending a search request to OpenSearch with the specified pipeline.

```bash
curl 'http://localhost:9200/demoindex0/_search?search_pipeline=remote_pipeline&pretty' --json ' {
  "query": {
    "match": {
      "text_representation": "armadillo"
    }
  }
}'
```

--------------------------------

### Run First RAG Pipeline

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/aryn-opensearch-bedrock-rag-example.ipynb

Executes a basic RAG pipeline. It sets up an embedder, constructs an OpenSearch query, retrieves documents, limits the context size, and then summarizes the data to answer a question. Dependencies include SentenceTransformerEmbedder, openSearch_client_args, get_knn_query, and summarize_data.

```python
context_size = 20
question = "What was common with incidents in Texas, and how does that differ from incidents in California?"

text_embedder = SentenceTransformerEmbedder(batch_size=10_000, model_name=model_name)
os_client_args = openSearch_client_args

os_query = get_knn_query(query_phrase=question, context=context, text_embedder=text_embedder)

docset = context.read.opensearch(index_name="aryn-rag-demo", query=os_query, os_client_args=os_client_args)
docset = docset.limit(context_size)

answer = summarize_data(
    question=question,
    result_description="Documents returned that can answer a quesiton about flight incidents",
    result_data=[docset],
    context=context,
    llm=llm,
)
print(answer)
```

--------------------------------

### Install Poppler on macOS

Source: https://github.com/aryn-ai/sycamore/blob/main/CONTRIBUTING.md

Installs the Poppler PDF rendering library on macOS using the Homebrew package manager. This is a dependency for some PDF processing functionalities in Sycamore.

```bash
brew install poppler

```

--------------------------------

### Install Poppler Utilities

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/text-embedding-3-small_greedy-text-element-merger_opensearch.ipynb

Installs the Poppler utilities, which are required for PDF processing within the Sycamore library. This is a system-level dependency for handling PDF files.

```python
!apt-get install poppler-utils
```

--------------------------------

### Import necessary libraries

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/financial-docs-10k-example.ipynb

Imports the boto3 library for AWS S3 interaction and specific functions from the aryn_sdk for document partitioning and table conversion.

```python
# Run pip install aryn-sdk first
import boto3
from aryn_sdk.partition import partition_file, tables_to_pandas
```

--------------------------------

### Install Poppler Utilities

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/text-embedding-3-small_greedy-section-merger_duckdb.ipynb

Installs the Poppler utilities, which are required for PDF rendering and processing by Sycamore. This is a system-level dependency for handling PDF files.

```bash
!apt-get install poppler-utils
```

--------------------------------

### Initializing Sycamore Context and LLM

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/metadata-extraction.ipynb

Initializes the Sycamore execution context and sets up an OpenAI language model instance using the GPT-3.5-turbo model. It also initializes a HuggingFace tokenizer.

```python
s3_path = "s3://aryn-public/ntsb/"
partition_materialize_path = "s3://aryn-public/materialize/notebooks/partition-ntsb/2024-10-11"

llm = OpenAI(OpenAIModels.GPT_3_5_TURBO.value)
tokenizer = HuggingFaceTokenizer("thenlper/gte-small")

ctx = sycamore.init()
```

--------------------------------

### Configure API Keys (Optional)

Source: https://github.com/aryn-ai/sycamore/blob/main/notebooks/docprep/minilm-l6-v2_greedy-text-element-merger_duckdb.ipynb

Provides guidance on storing API keys, suggesting environment variables or configuration files. It includes a commented-out example for quick testing.

```python
# It's best to store API keys in a configuration file or set them as environment variables.
# For quick testing, you can define them here:
#
# os.environ["ARYN_API_KEY"] = "YOUR_ARYN_API_KEY"
```