### Initial Docker Compose Setup Source: https://github.com/morphik-org/morphik-core/blob/main/DOCKER.md Builds all required containers, downloads AI models, initializes the database, and starts all services for the first-time setup. ```bash docker compose up --build ``` -------------------------------- ### Run Morphik SDK Example Usage Scripts Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/morphik/tests/README.md Execute example scripts to demonstrate basic SDK usage. Use the --async flag to run the asynchronous version of the example. ```bash python example_usage.py ``` ```bash python example_usage.py --async ``` -------------------------------- ### Build and Start Production Application Source: https://github.com/morphik-org/morphik-core/blob/main/ee/ui-component/README.md Build the project for production and start the production server. ```bash npm run build npm start ``` -------------------------------- ### Start and Stop Docker Services Source: https://github.com/morphik-org/morphik-core/blob/main/DOCKER.md Commands to start all services after the initial setup and to stop them when not in use. ```bash docker compose up ``` ```bash docker compose down ``` -------------------------------- ### Run Morphik Example Usage Script Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/README.md Execute the example usage script to demonstrate basic SDK functionality, with options for synchronous and asynchronous execution. ```bash # Run synchronous example python -m morphik.tests.example_usage # Run asynchronous example python -m morphik.tests.example_usage --async ``` -------------------------------- ### Install Project Dependencies Source: https://github.com/morphik-org/morphik-core/blob/main/ee/ui-component/README.md Install all necessary dependencies for development using npm. ```bash npm install ``` -------------------------------- ### Install @morphik/ui Source: https://github.com/morphik-org/morphik-core/blob/main/ee/ui-component/README.md Install the @morphik/ui package using npm. ```bash npm install @morphik/ui ``` -------------------------------- ### Install Python Dependencies Source: https://github.com/morphik-org/morphik-core/blob/main/CLAUDE.md Installs project dependencies using uv. Ensure uv is installed and configured for your environment. ```bash uv sync ``` -------------------------------- ### Install Morphik Python Client Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/README.md Install the Morphik Python client using pip. ```bash pip install morphik ``` -------------------------------- ### Clone Repository and Navigate Source: https://github.com/morphik-org/morphik-core/blob/main/DOCKER.md Clone the Morphik Core repository and change into the project directory to begin setup. ```bash git clone https://github.com/morphik-org/morphik-core.git cd morphik-core ``` -------------------------------- ### Install Frontend Dependencies Source: https://github.com/morphik-org/morphik-core/blob/main/CLAUDE.md Installs Node.js dependencies for the frontend project using npm. Navigate to the frontend directory first. ```bash cd ee/ui-component npm install ``` -------------------------------- ### Run Frontend Development Server Source: https://github.com/morphik-org/morphik-core/blob/main/CLAUDE.md Starts the Next.js development server for the frontend. This command should be run from the `ee/ui-component` directory. ```bash npm run dev # Development server ``` -------------------------------- ### Start Python Development Server Source: https://github.com/morphik-org/morphik-core/blob/main/CLAUDE.md Launches the FastAPI development server. Use `start_server.py` for direct startup or `./start-dev.sh` for a Dockerized environment. ```bash python start_server.py # Direct startup ./start-dev.sh # Docker development environment ``` -------------------------------- ### Install Development Dependencies for Morphik Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/README.md Install the necessary development dependencies for running tests and contributing to the Morphik project. ```bash pip install -r test_requirements.txt ``` -------------------------------- ### Docker Development Environment Source: https://github.com/morphik-org/morphik-core/blob/main/CLAUDE.md Manages the Docker development environment. Use `./start-dev.sh` for hot-reloading or `docker compose` for standard setup. ```bash # Development environment with hot reload ./start-dev.sh # Standard Docker setup docker compose up --build docker compose down docker compose down -v # Reset all data ``` -------------------------------- ### Environment Variables for Customization Source: https://github.com/morphik-org/morphik-core/blob/main/DOCKER.md Example of a .env file for customizing Morphik settings. Key variables include JWT_SECRET_KEY for security, OPENAI_API_KEY if using OpenAI, and HOST/PORT for network configuration. ```bash JWT_SECRET_KEY=your-secure-key-here OPENAI_API_KEY=sk-... HIST=0.0.0.0 PORT=8000 ``` -------------------------------- ### Build and Upload Python Package to PyPI Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/PUBLISH.md Installs necessary build tools, cleans previous builds, builds the package, checks its integrity, and uploads it to PyPI. Ensure you have activated the correct Python environment and have the necessary PyPI credentials. ```bash # ensure you've activated the correct python environment pip install build twine rm -rf dist uv run python -m build uv run twine check dist/* uv run twine upload dist/* ``` -------------------------------- ### Query Data with Python SDK Source: https://github.com/morphik-org/morphik-core/blob/main/README.md Use the Morphik Python SDK to search and query your data. This example demonstrates a natural language query. ```python morphik.query("What's the height of screw 14-A in the chair assembly instructions?") ``` -------------------------------- ### Migrate Authentication Columns Source: https://github.com/morphik-org/morphik-core/blob/main/README.md Run this script to optimize the authentication system for faster query performance. This is required for existing installations before launching Morphik. ```bash python scripts/migrate_auth_columns_complete.py --postgres-uri "postgresql+asyncpg://user:pass@host:port/db" ``` -------------------------------- ### Get Document by Filename Source: https://context7.com/morphik-org/morphik-core/llms.txt Retrieve a document by its original filename, optionally scoped to a folder or user. ```bash curl "http://localhost:8000/documents/filename/annual_report.pdf?folder_name=reports%2F2024" \ -H "Authorization: Bearer " ``` -------------------------------- ### Get Folder Details with Statistics Source: https://context7.com/morphik-org/morphik-core/llms.txt Use this endpoint to retrieve detailed metadata for folders, including document counts and paginated lists. Requires a Bearer token for authorization. ```bash curl -X POST http://localhost:8000/folders/details \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '{ "identifiers": ["finance", "q1-reports"], "include_document_count": true, "include_status_counts": true, "include_documents": true, "document_limit": 10, "document_fields": ["external_id", "filename", "metadata.year"] }' ``` -------------------------------- ### Manage Document Summary using cURL Source: https://context7.com/morphik-org/morphik-core/llms.txt Retrieve or upsert (create/update with versioning) a text summary for a document. Use PUT to write or update, and GET to read the latest summary. ```bash # Write a summary curl -X PUT http://localhost:8000/documents/doc_abc123/summary \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '{"content": "This document covers Morphik product overview, key features, and pricing.", "versioning": true}' ``` ```bash # Read the latest summary curl http://localhost:8000/documents/doc_abc123/summary \ -H "Authorization: Bearer " ``` ```json # Response: # {"content": "This document covers...", "storage_key": "...", "version": 1, "updated_at": "2025-01-15T12:05:00"} ``` -------------------------------- ### Stop Services with Compose Profiles Source: https://github.com/morphik-org/morphik-core/blob/main/DOCKER.md Example of stopping services when a specific compose profile, like 'ui', is enabled. Ensure to include the profile name when stopping services to properly manage associated volumes and containers. ```bash docker compose --profile ui down --volumes --remove-orphans ``` -------------------------------- ### Initialize Morphik Python Client (Sync/Async) Source: https://context7.com/morphik-org/morphik-core/llms.txt Demonstrates initialization of both synchronous (`Morphik`) and asynchronous (`AsyncMorphik`) clients. Supports local development and cloud connections with authentication. ```python from morphik import Morphik from morphik.async_ import AsyncMorphik # Sync — local dev (no auth) client = Morphik() # http://localhost:8000 # Sync — cloud with auth client = Morphik("morphik://owner_id:token@api.morphik.ai", timeout=60) # Async — cloud with auth, used as context manager async with AsyncMorphik("morphik://owner_id:token@api.morphik.ai") as aclient: health = await aclient.get_health() print(health.status) # "healthy" # Sync context manager with Morphik("morphik://owner_id:token@api.morphik.ai") as client: ping = client.ping() print(ping) # {"status": "ok", "message": "Server is running"} ``` -------------------------------- ### Folder Management with Morphik Client Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/README.md Shows how to create, move, and rename folders, and how to scope queries to specific paths with folder depth. ```python # Create a nested folder (parents are auto-created server-side) folder = db.create_folder(full_path="/projects/alpha/specs", description="Specs folder") # Move or rename folder paths moved = db.move_folder("/projects/alpha/specs", "/projects/archive/specs") renamed = moved.rename("specs-v2") # Scope queries to a path and include descendants with folder_depth=-1 chunks = folder.retrieve_chunks(query="design notes", folder_depth=-1) docs = db.list_documents(folder_name="/projects/alpha", folder_depth=-1) ``` -------------------------------- ### Asynchronous Morphik Client Usage Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/README.md Demonstrates initializing the asynchronous Morphik client and performing operations like document ingestion and RAG queries within an async context. ```python import asyncio from morphik.async_ import AsyncMorphik async def main(): # Initialize async client - connects to localhost:8000 by default async with AsyncMorphik() as db: # You can also use a direct HTTP(S) base URL for self-hosted deployments # async with AsyncMorphik("http://morphik:8000") as db: # Or with authentication URI (for production) # async with AsyncMorphik("morphik://owner_id:token@api.morphik.ai") as db: # Ingest a text document doc = await db.ingest_text( content="Your document content", metadata={"title": "Example Document"} ) doc_query = await db.query_document( file="path/to/document.pdf", prompt="Extract the parties and effective date.", ingestion_options={"ingest": True, "metadata": {"source": "contracts"}} ) print(doc_query.structured_output) # Query with RAG response = await db.query( query="Summarize the key points in the document", ) print(response.completion) # Run the async function asyncio.run(main()) ``` -------------------------------- ### Get Document Metadata Source: https://context7.com/morphik-org/morphik-core/llms.txt Retrieve full metadata for a document by its external ID. ```bash curl http://localhost:8000/documents/doc_abc123 \ -H "Authorization: Bearer " ``` -------------------------------- ### Get Document Status Source: https://context7.com/morphik-org/morphik-core/llms.txt Poll the processing status of a document, which can be 'processing', 'completed', or 'failed'. ```APIDOC ## GET /documents/{document_id}/status ### Description Poll the processing status of a document (`processing`, `completed`, `failed`). ### Method GET ### Endpoint /documents/{document_id}/status ### Parameters #### Path Parameters - **document_id** (string) - Required - The external ID of the document whose status is to be retrieved. ### Request Example ```bash curl http://localhost:8000/documents/doc_abc123 \ -H "Authorization: Bearer " ``` ### Response #### Success Response (200) - **document_id** (string) - The external ID of the document. - **status** (string) - The current processing status of the document (e.g., `processing`, `completed`, `failed`). - **filename** (string) - The filename of the document. - **progress** (object) - Detailed progress information if the document is still processing. - **current_step** (integer) - The current step in the processing pipeline. - **total_steps** (integer) - The total number of steps in the processing pipeline. - **step_name** (string) - The name of the current processing step. - **percentage** (integer) - The completion percentage of the current step. - **error** (string) - An error message if the document processing failed. - **created_at** (string) - Timestamp when the document processing was initiated. - **updated_at** (string) - Timestamp when the document processing was last updated. #### Response Example ```json # While processing: {"document_id": "doc_abc123", "status": "processing", "filename": "annual_report.pdf", "progress": {"current_step": 2, "total_steps": 5, "step_name": "Chunking", "percentage": 40}} # On completion: {"document_id": "doc_abc123", "status": "completed", "filename": "annual_report.pdf", "created_at": "2025-01-15T12:00:00", "updated_at": "2025-01-15T12:01:30"} # On failure: {"document_id": "doc_abc123", "status": "failed", "error": "Parser timeout"} ``` ``` -------------------------------- ### Get Document Source: https://context7.com/morphik-org/morphik-core/llms.txt Retrieve the full metadata for a specific document using its external ID. ```APIDOC ## GET /documents/{document_id} ### Description Retrieve full metadata for a document by its external ID. ### Method GET ### Endpoint /documents/{document_id} ### Parameters #### Path Parameters - **document_id** (string) - Required - The external ID of the document to retrieve. ### Request Example ```bash curl http://localhost:8000/documents/doc_abc123 \ -H "Authorization: Bearer " ``` ### Response #### Success Response (200) - **external_id** (string) - The unique external identifier for the document. - **filename** (string) - The original filename of the document. - **content_type** (string) - The MIME type of the document's content. - **metadata** (object) - Custom metadata associated with the document. - **system_metadata** (object) - System-generated metadata, including status and timestamps. - **chunk_ids** (array of strings) - Identifiers for the chunks the document was split into. - **page_count** (integer) - The total number of pages in the document. - **folder_name** (string) - The name of the folder the document belongs to. #### Response Example ```json { "external_id": "doc_abc123", "filename": "morphik_overview.txt", "content_type": "text/plain", "metadata": {"category": "product", "version": 1.0}, "system_metadata": {"status": "completed", "created_at": "...", "updated_at": "..."}, "chunk_ids": ["chunk_1", "chunk_2", "chunk_3"], "page_count": 1, "folder_name": "product-docs" } ``` ``` -------------------------------- ### Get Document Status Source: https://context7.com/morphik-org/morphik-core/llms.txt Poll the processing status of a document ('processing', 'completed', 'failed'). ```bash curl http://localhost:8000/documents/doc_abc123/status \ -H "Authorization: Bearer " ``` -------------------------------- ### Synchronous Morphik Client Usage Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/README.md Demonstrates initializing the synchronous Morphik client and performing common operations like document ingestion, file ingestion, document querying, chunk retrieval, and RAG queries. ```python from morphik import Morphik # Initialize client - connects to localhost:8000 by default db = Morphik() # You can also use a direct HTTP(S) base URL for self-hosted deployments # db = Morphik("http://morphik:8000") # Or with authentication URI (for production) # db = Morphik("morphik://owner_id:token@api.morphik.ai") # Ingest a text document doc = db.ingest_text( content="Your document content", metadata={"title": "Example Document"} ) # Ingest a file doc = db.ingest_file( file="path/to/document.pdf", metadata={"category": "reports"} ) # Run a Morphik On-the-Fly document query doc_query = db.query_document( file="path/to/document.pdf", prompt="Extract the parties and effective date.", ingestion_options={"ingest": True, "metadata": {"source": "contracts"}} ) print(doc_query.structured_output) # Retrieve relevant chunks chunks = db.retrieve_chunks( query="Your search query", filters={"category": "reports"} ) # Query with RAG response = db.query( query="Summarize the key points in the document", filters={"category": "reports"} ) print(response.completion) ``` -------------------------------- ### Frontend Build and Quality Checks Source: https://github.com/morphik-org/morphik-core/blob/main/CLAUDE.md Commands for building the frontend application and performing quality checks like linting and formatting. ```bash npm run build # Production build npm run build:package # Package build npm run lint # ESLint npm run format # Prettier formatting npm run format:check # Check formatting ``` -------------------------------- ### Get Document by Filename Source: https://context7.com/morphik-org/morphik-core/llms.txt Retrieve a document by its original filename. This search can be optionally scoped to a specific folder or user. ```APIDOC ## GET /documents/filename/{filename} ### Description Retrieve a document by its original filename, optionally scoped to a folder or user. ### Method GET ### Endpoint /documents/filename/{filename} ### Parameters #### Path Parameters - **filename** (string) - Required - The original filename of the document to retrieve. #### Query Parameters - **folder_name** (string) - Optional - The name of the folder to scope the search within. ### Request Example ```bash curl "http://localhost:8000/documents/filename/annual_report.pdf?folder_name=reports%2F2024" \ -H "Authorization: Bearer " ``` ``` -------------------------------- ### Folder Scoping for Ingestion and Retrieval Source: https://context7.com/morphik-org/morphik-core/llms.txt Demonstrates how to create and manage folders, and scope ingestion, retrieval, and RAG queries to a specific folder. Operations on a `Folder` object are automatically scoped. ```python from morphik import Morphik client = Morphik() # Create nested folder (ancestors auto-created) folder = client.create_folder( name="contracts", full_path="/legal/2025/contracts", description="Active contracts for 2025", ) # Ingest scoped to this folder doc = folder.ingest_file("master_agreement.pdf", metadata={"client": "Acme", "type": "MSA"}, use_colpali=True) # Retrieve scoped to this folder chunks = folder.retrieve_chunks(query="liability cap and indemnification", k=4) # RAG query scoped to this folder response = folder.query( query="What are the payment terms across all contracts?", k=8, use_reranking=True, ) print(response.completion) # List documents in folder docs = folder.list_documents(completed_only=True, sort_by="filename") # Move folder folder.move("/legal/archive/2025/contracts") # Get summary for the folder folder.upsert_summary("Contains 14 active MSAs for FY2025 clients.", versioning=True) summary = folder.get_summary() print(summary.content) # Access folder info print(folder.full_path, folder.depth, folder.id) ``` -------------------------------- ### Get Document Download URL Source: https://context7.com/morphik-org/morphik-core/llms.txt Generate a presigned URL to download the original file. This provides temporary access to the document file. ```APIDOC ## GET /documents/{document_id}/download_url ### Description Generate a presigned URL to download the original file. ### Method GET ### Endpoint /documents/{document_id}/download_url ### Parameters #### Path Parameters - **document_id** (string) - Required - The ID of the document to get the download URL for. #### Query Parameters - **expires_in** (integer) - Optional - The duration in seconds for which the URL will be valid. ``` -------------------------------- ### Check Server Health - GET /ping Source: https://context7.com/morphik-org/morphik-core/llms.txt Use this endpoint to confirm if the Morphik server is running and responsive. It returns a simple status. ```bash curl http://localhost:8000/ping # Response: # {"status": "ok", "message": "Server is running"} ``` -------------------------------- ### Run Knowledge Graph Creation Source: https://github.com/morphik-org/morphik-core/blob/main/evaluations/Science graphs (SciER)/README.md Execute the script to generate a knowledge graph using the model specified in `morphik.toml`. The script outputs a graph name upon completion. ```bash python scier_evaluation.py --model-name gpt4o ``` -------------------------------- ### Create Folder using cURL Source: https://context7.com/morphik-org/morphik-core/llms.txt Create a folder, with automatic ancestor creation for nested paths, to scope documents and operations. Provide a name, full path, and optional description. ```bash curl -X POST http://localhost:8000/folders \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '{"name": "q1-reports", "full_path": "/finance/2025/q1-reports", "description": "Q1 2025 financial reports"}' ``` ```json # Response: # { # "id": "folder_def456", # "name": "q1-reports", # "full_path": "/finance/2025/q1-reports", # "parent_id": "folder_parent789", # "depth": 3, # "description": "Q1 2025 financial reports" # } ``` -------------------------------- ### Analyze Evaluation Results Source: https://github.com/morphik-org/morphik-core/blob/main/evaluations/custom_eval/README.md Perform analysis on the judged evaluation results. Options include using custom golden answers or showing more examples. ```bash # Simple accuracy analysis python analyze_eval.py morphik_results_judged.csv # With custom golden answers file python analyze_eval.py results.csv --golden my_questions.csv # Show more examples python analyze_eval.py results.csv --show-examples 5 ``` -------------------------------- ### On-the-Fly Document Query Source: https://context7.com/morphik-org/morphik-core/llms.txt Analyze a document inline without persistent storage. Optionally enforce structured JSON output with a schema or trigger a follow-up ingestion. ```bash curl -X POST http://localhost:8000/ingest/document/query \ -H "Authorization: Bearer " \ -F "file=@invoice.pdf" \ -F "prompt=Extract vendor name, total amount, and invoice date" \ -F 'schema={"type":"object","properties":{"vendor":{"type":"string"},"total":{"type":"number"},"date":{"type":"string"}},"required":["vendor","total","date"]}' \ -F 'ingestion_options={"ingest":true,"use_colpali":true,"folder_name":"invoices","metadata":{"source":"accounts-payable"}}' ``` -------------------------------- ### Run Morphik Tests Source: https://github.com/morphik-org/morphik-core/blob/main/sdks/python/README.md Commands for running tests, including options for running all tests, specific modules, skipping live tests, and specifying a custom server URL. ```bash # Run all tests (requires a running Morphik server) pytest morphik/tests/ -v # Run specific test modules pytest morphik/tests/test_sync.py -v pytest morphik/tests/test_async.py -v # Skip tests if you don't have a running server SKIP_LIVE_TESTS=1 pytest morphik/tests/ -v # Specify a custom server URL for tests MORPHIK_TEST_URL=http://custom-server:8000 pytest morphik/tests/ -v ``` -------------------------------- ### Morphik Client Initialization Source: https://context7.com/morphik-org/morphik-core/llms.txt Initializes the Morphik client for synchronous or asynchronous operations. Supports local development and cloud connections with authentication. ```APIDOC ## Morphik Client Initialization ### Description The SDK provides `Morphik` (sync) and `AsyncMorphik` (async) clients. Both support folder scoping via `create_folder()` / `get_folder_by_name()` and user scoping via `signin()`. ### Code Examples ```python from morphik import Morphik from morphik.async_ import AsyncMorphik # Sync — local dev (no auth) client = Morphik() # http://localhost:8000 # Sync — cloud with auth client = Morphik("morphik://owner_id:token@api.morphik.ai", timeout=60) # Async — cloud with auth, used as context manager async with AsyncMorphik("morphik://owner_id:token@api.morphik.ai") as aclient: health = await aclient.get_health() print(health.status) # "healthy" # Sync context manager with Morphik("morphik://owner_id:token@api.morphik.ai") as client: ping = client.ping() print(ping) # {"status": "ok", "message": "Server is running"} ``` ``` -------------------------------- ### Batch Get Documents and Chunks Source: https://context7.com/morphik-org/morphik-core/llms.txt Fetch multiple documents or specific chunks from documents efficiently in a single API call. Useful for reducing network overhead when retrieving many items. ```python from morphik import Morphik from morphik.models import ChunkSource client = Morphik() # Fetch multiple documents in one call docs = client.batch_get_documents( document_ids=["doc_001", "doc_002", "doc_003"], folder_name="legal/contracts", ) # Fetch specific chunks by document + chunk number chunks = client.batch_get_chunks( sources=[ ChunkSource(document_id="doc_001", chunk_number=5), ChunkSource(document_id="doc_001", chunk_number=6), ChunkSource(document_id="doc_002", chunk_number=12), ], use_colpali=True, output_format="base64", ) for chunk in chunks: print(f"Doc {chunk.document_id} chunk {chunk.chunk_number}: {chunk.content[:80]}...") ``` -------------------------------- ### Evaluate Created Knowledge Graph Source: https://github.com/morphik-org/morphik-core/blob/main/evaluations/Science graphs (SciER)/README.md Run the evaluation script, providing the name of the previously created knowledge graph to assess its performance. ```bash python evaluate_result.py --graph-name scier_gpt4o_12345678 ``` -------------------------------- ### Get Document Download URL using cURL Source: https://context7.com/morphik-org/morphik-core/llms.txt Generate a presigned URL to download the original file of a document. The `expires_in` parameter controls the URL's validity period in seconds. ```bash curl "http://localhost:8000/documents/doc_xyz789/download_url?expires_in=7200" \ -H "Authorization: Bearer " ``` ```json # Response: # { # "document_id": "doc_xyz789", # "filename": "annual_report.pdf", # "content_type": "application/pdf", # "download_url": "https://s3.amazonaws.com/...?X-Amz-Expires=7200...", # "expires_in": 7200 # } ``` -------------------------------- ### Detailed Server Health Check - GET /health Source: https://context7.com/morphik-org/morphik-core/llms.txt Provides a comprehensive health status of the Morphik server and its connected services, including database, vector store, and storage. Useful for diagnosing connectivity issues. ```bash curl http://localhost:8000/health # Response: # { # "status": "healthy", # "timestamp": "2025-01-15T12:00:00+00:00", # "services": [ # {"name": "postgresql", "status": "healthy", "message": "Database connection successful", "response_time_ms": 2.4}, # {"name": "redis", "status": "healthy", "message": "Redis connection successful", "response_time_ms": 0.8}, # {"name": "pgvector", "status": "healthy", "message": "Vector store initialized", "response_time_ms": 1.1}, # {"name": "storage", "status": "healthy", "message": "Local storage at ./storage is accessible", "response_time_ms": 0.2}, # {"name": "colpali", "status": "healthy", "message": "ColPali vector store initialized","response_time_ms": 0.5} # ] # } ``` -------------------------------- ### Upload and Ingest a File - POST /ingest/file Source: https://context7.com/morphik-org/morphik-core/llms.txt Uploads a binary file (e.g., PDF, DOCX, image) for processing. Uses multipart form data and supports optional metadata and ColPali embedding. ```bash curl -X POST http://localhost:8000/ingest/file \ -H "Authorization: Bearer " \ -F "file=@/path/to/annual_report.pdf" \ -F 'metadata={"department": "finance", "year": 2024}' \ -F 'metadata_types={"year": "int"}' \ -F "use_colpali=true" \ -F "folder_name=reports/2024" \ -F "end_user_id=analyst-7" # Response: # { # "external_id": "doc_xyz789", # "filename": "annual_report.pdf", # "content_type": "application/pdf", # "metadata": {"department": "finance", "year": 2024}, # "system_metadata": {"status": "processing"}, # "folder_name": "reports/2024" # } ``` -------------------------------- ### Ingest File with Python SDK Source: https://github.com/morphik-org/morphik-core/blob/main/README.md Use the Morphik Python SDK to ingest a file. Ensure you have initialized the Morphik client with your Morphik URI. ```python from morphik import Morphik morphik = Morphik("") morphik.ingest_file("path/to/your/super/complex/file.pdf") ``` -------------------------------- ### List Documents with Filters Source: https://context7.com/morphik-org/morphik-core/llms.txt Flexible document listing with metadata filters, pagination, sorting, folder scoping, and aggregation. ```bash # Filter by metadata + folder scope + pagination curl -X POST http://localhost:8000/documents \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '{ "document_filters": { "$and": [ {"department": "finance"}, {"year": {"$gte": 2023}} ] }, "skip": 0, "limit": 25, "sort_by": "updated_at", "sort_direction": "desc", "include_total_count": true, "include_status_counts": true, "completed_only": true, "fields": ["external_id", "filename", "metadata.year"] }' \ --data-urlencode "folder_name=reports/2024" ``` -------------------------------- ### Manually Format Python Code Source: https://github.com/morphik-org/morphik-core/blob/main/scripts/README.md Run this script to manually format all Python files in the project using isort, black, and ruff. It applies the same formatting rules as the pre-commit hook. ```bash ./scripts/format.sh ``` -------------------------------- ### Async Morphik for Full Async Support Source: https://context7.com/morphik-org/morphik-core/llms.txt Utilize the AsyncMorphik client for non-blocking I/O operations. All methods mirror the synchronous client but require `await` and are run within an asyncio event loop. ```python import asyncio from morphik.async_ import AsyncMorphik from morphik.models import ChunkSource async def main(): async with AsyncMorphik("morphik://owner:token@api.morphik.ai") as client: # Ingest doc = await client.ingest_file("report.pdf", use_colpali=True) completed = await client.wait_for_document_completion(doc.external_id, timeout_seconds=300) # Folder scope (async) folder = await client.create_folder(name="reports", full_path="/finance/reports") scoped_doc = await folder.ingest_text("Revenue up 18%", filename="summary.txt") # User scope (async) user = client.signin("user-bob") user_doc = await user.ingest_file("bob_notes.pdf") # Retrieve and query chunks = await client.retrieve_chunks( query="revenue growth drivers", k=5, use_colpali=True, folder_name="/finance/reports" ) response = await client.query( query="What drove revenue growth?", k=8, use_reranking=True, folder_name="/finance/reports" ) print(response.completion) # Batch batch = await client.batch_get_chunks( [ChunkSource(document_id=doc.external_id, chunk_number=0)], use_colpali=True ) asyncio.run(main()) ``` -------------------------------- ### List Folders Source: https://context7.com/morphik-org/morphik-core/llms.txt List all folders accessible to the authenticated user. This provides an overview of the folder structure. ```APIDOC ## GET /folders ### Description List all folders accessible to the authenticated user. ### Method GET ### Endpoint /folders ``` -------------------------------- ### Test Different Models Source: https://github.com/morphik-org/morphik-core/blob/main/evaluations/Science graphs (SciER)/README.md To compare models, modify the `morphik.toml` file, repeat the graph creation and evaluation steps with a new model name, and then compare the resulting metrics. ```bash # Change the model in morphik.toml # Repeat steps 2-3 with a different --model-name # Compare the resulting metrics and visualizations ``` -------------------------------- ### View Docker Compose Logs Source: https://github.com/morphik-org/morphik-core/blob/main/DOCKER.md Commands to view logs for all running Docker Compose services or for a specific service like 'morphik', 'postgres', or 'ollama'. Useful for diagnosing startup or runtime issues. ```bash docker compose logs docker compose logs morphik docker compose logs postgres docker compose logs ollama ``` -------------------------------- ### AsyncMorphik - Full Async Support Source: https://context7.com/morphik-org/morphik-core/llms.txt Utilize asynchronous operations for all Morphik methods using the AsyncMorphik client. ```APIDOC ## AsyncMorphik All methods available on the `Morphik` client have an asynchronous equivalent on `AsyncMorphik`. Operations are performed using `await`. ### Example Usage ```python import asyncio from morphik.async_ import AsyncMorphik from morphik.models import ChunkSource async def main(): async with AsyncMorphik("morphik://owner:token@api.morphik.ai") as client: # Ingest file asynchronously doc = await client.ingest_file("report.pdf", use_colpali=True) completed = await client.wait_for_document_completion(doc.external_id, timeout_seconds=300) # Create folder and ingest text asynchronously folder = await client.create_folder(name="reports", full_path="/finance/reports") scoped_doc = await folder.ingest_text("Revenue up 18%", filename="summary.txt") # User scope operations asynchronously user = client.signin("user-bob") user_doc = await user.ingest_file("bob_notes.pdf") # Retrieve chunks asynchronously chunks = await client.retrieve_chunks( query="revenue growth drivers", k=5, use_colpali=True, folder_name="/finance/reports" ) # Query asynchronously response = await client.query( query="What drove revenue growth?", k=8, use_reranking=True, folder_name="/finance/reports" ) print(response.completion) # Batch get chunks asynchronously batch = await client.batch_get_chunks( [ChunkSource(document_id=doc.external_id, chunk_number=0)], use_colpali=True ) asyncio.run(main()) ``` ``` -------------------------------- ### List Folders using cURL Source: https://context7.com/morphik-org/morphik-core/llms.txt List all folders accessible to the authenticated user. The response includes folder details such as ID, name, full path, and depth. ```bash curl http://localhost:8000/folders \ -H "Authorization: Bearer " ``` ```json # Response: # [ # {"id": "folder_001", "name": "finance", "full_path": "/finance", "depth": 1, ...}, # {"id": "folder_002", "name": "2025", "full_path": "/finance/2025", "depth": 2, ...}, # {"id": "folder_def456", "name": "q1-reports","full_path": "/finance/2025/q1-reports","depth":3,... # ] ``` -------------------------------- ### Morphik.ingest_files() and Morphik.ingest_directory() Source: https://context7.com/morphik-org/morphik-core/llms.txt Provides methods for batch ingestion of multiple files, either from a list of files with associated metadata or from an entire directory. ```APIDOC ## Batch Ingestion ### Description Ingests multiple documents in batches, either from a list of files with per-file metadata or from an entire directory. ### Methods #### `ingest_files()` - **Parameters**: - **files** (list[str]) - Required - A list of file paths to ingest. - **metadata** (list[dict]) - Optional - A list of dictionaries, where each dictionary contains metadata for the corresponding file in the `files` list. - **use_colpali** (bool) - Optional - Whether to use colpali for processing. Defaults to False. - **parallel** (bool) - Optional - Whether to process files in parallel. Defaults to False. - **Returns**: - list[Document] - A list of ingested document objects. #### `ingest_directory()` - **Parameters**: - **directory** (str) - Required - The path to the directory to ingest. - **recursive** (bool) - Optional - Whether to ingest files recursively from subdirectories. Defaults to False. - **pattern** (str) - Optional - A glob pattern to filter files (e.g., "*.pdf"). - **metadata** (dict) - Optional - A dictionary of metadata to apply to all files in the directory. - **use_colpali** (bool) - Optional - Whether to use colpali for processing. Defaults to False. - **Returns**: - list[Document] - A list of ingested document objects. ### Request Example ```python # Batch with per-file metadata docs = client.ingest_files( files=["report_jan.pdf", "report_feb.pdf", "report_mar.pdf"], metadata=[ {"month": "Jan", "year": 2025}, {"month": "Feb", "year": 2025}, {"month": "Mar", "year": 2025}, ], use_colpali=True, parallel=True, ) print(f"Ingested {len(docs)} documents") # Entire directory, only PDFs, recursively all_docs = client.ingest_directory( directory="/data/documents", recursive=True, pattern="*.pdf", metadata={"source": "data-lake"}, use_colpali=True, ) ``` ``` -------------------------------- ### Create Folder Source: https://context7.com/morphik-org/morphik-core/llms.txt Create a folder (with automatic ancestor creation for nested paths) to scope documents and operations. This helps in organizing documents hierarchically. ```APIDOC ## POST /folders ### Description Create a folder (with automatic ancestor creation for nested paths) to scope documents and operations. ### Method POST ### Endpoint /folders ### Parameters #### Request Body - **name** (string) - Required - The name of the folder. - **full_path** (string) - Optional - The full path for the folder, including ancestors. - **description** (string) - Optional - A description for the folder. ``` -------------------------------- ### Morphik Configuration File Source: https://github.com/morphik-org/morphik-core/blob/main/DOCKER.md Default TOML configuration for Morphik, specifying API host and port, Ollama provider details for completion and embedding, and database/storage settings. Ensure '0.0.0.0' is used for the API host when running in Docker. ```toml [api] host = "0.0.0.0" port = 8000 [completion] provider = "ollama" model_name = "llama3.2" base_url = "http://ollama:11434" [embedding] provider = "ollama" model_name = "nomic-embed-text" base_url = "http://ollama:11434" [database] provider = "postgres" [vector_store] provider = "pgvector" [storage] provider = "local" storage_path = "/app/storage" ```