### Install Dependencies with uv (Bash)

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Install project dependencies using uv after activating the virtual environment. Also runs the crawl4ai setup command.

```bash
uv pip install -e .
crawl4ai-setup
```

--------------------------------

### Install uv (Bash)

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Install the uv package manager if it is not already installed. This is a prerequisite for direct installation.

```bash
pip install uv
```

--------------------------------

### Hallucination Report Generation Example

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

An example demonstrating how to instantiate the HallucinationReporter and use its generate_comprehensive_report method to obtain a detailed hallucination report. The example shows how to access specific parts of the report, such as detected hallucinations, recommendations, and validation summary statistics.

```python
reporter = HallucinationReporter()
report = reporter.generate_comprehensive_report(validation_result)
# report['hallucinations_detected']: [...] 
# report['recommendations']: ["Fix method call...", ...]
# report['validation_summary']['hallucination_rate']: 0.15
```

--------------------------------

### AI Coding Assistant with Code Examples Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Configure these environment variables for an AI coding assistant that includes code examples. USE_CONTEXTUAL_EMBEDDINGS and USE_AGENTIC_RAG should be true.

```env
USE_CONTEXTUAL_EMBEDDINGS=true
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=true
USE_RERANKING=true
USE_KNOWLEDGE_GRAPH=false
```

--------------------------------

### Generate Code Example Summary (Python)

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Generates a concise summary for a code example by leveraging context before and after the code. It uses the OpenAI API with a model specified by the MODEL_CHOICE environment variable. Falls back to a generic message if generation fails.

```python
summary = generate_code_example_summary(
    code="async def authenticate(user, pwd):\n    return await verify(user, pwd)",
    context_before="Here's how to authenticate users:",
    context_after="This pattern ensures secure authentication."
)
# Returns: "Asynchronous authentication function that verifies user credentials..."
```

--------------------------------

### Neo4j Docker Setup

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Command to run Neo4j using Docker. Ensure Neo4j is running if USE_KNOWLEDGE_GRAPH is enabled.

```bash
docker run -d --name neo4j \
  -p 7687:7687 -p 7474:7474 \
  -e NEO4J_AUTH=neo4j/password \
  neo4j:latest
```

--------------------------------

### Example Usage of DirectNeo4jExtractor

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

Demonstrates how to instantiate the DirectNeo4jExtractor, initialize the connection, and analyze a GitHub repository. The repository's code structure will be stored in Neo4j upon successful analysis.

```python
extractor = DirectNeo4jExtractor("bolt://localhost:7687", "neo4j", "password")
await extractor.initialize()

await extractor.analyze_repository("https://github.com/pydantic/pydantic-ai.git")
# Repository now available for validation
```

--------------------------------

### Basic RAG Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Use this profile for initial setup, fast crawling, and basic semantic search. It minimizes features for quick indexing.

```env
# Server
HOST=0.0.0.0
PORT=8051
TRANSPORT=sse

# OpenAI
OPENAI_API_KEY=sk-...
MODEL_CHOICE=gpt-3.5-turbo

# Supabase
SUPABASE_URL=https://myproject.supabase.co
SUPABASE_SERVICE_KEY=eyJhbGci...

# RAG Strategies (Disabled)
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=false
USE_AGENTIC_RAG=false
USE_RERANKING=false
USE_KNOWLEDGE_GRAPH=false
```

--------------------------------

### Add Code Examples to Supabase (Python)

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Adds code examples to the Supabase `code_examples` table in batches. It handles deletion of existing examples for the same URLs, creates embeddings, and inserts data with retry logic. Supports batch insertion and falls back to individual insertion if batching fails.

```python
def add_code_examples_to_supabase(
    client: Client,
    urls: List[str],
    chunk_numbers: List[int],
    code_examples: List[str],
    summaries: List[str],
    metadatas: List[Dict[str, Any]],
    batch_size: int = 20
)
```

--------------------------------

### Fast Basic RAG Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Use this configuration for a fast, basic RAG setup. Both USE_AGENTIC_RAG and USE_RERANKING should be false.

```env
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=false
USE_RERANKING=false
USE_KNOWLEDGE_GRAPH=false
```

--------------------------------

### Get Supabase Client

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Initializes and returns a configured Supabase client instance. Requires SUPABASE_URL and SUPABASE_SERVICE_KEY environment variables to be set.

```python
def get_supabase_client() -> Client:
    # Implementation omitted for brevity
    pass

# Example Usage:
# client = get_supabase_client()
# Use client to interact with Supabase tables
```

--------------------------------

### Search Code Examples

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Searches for code examples. Requires `USE_AGENTIC_RAG=true`. Supports filtering by source ID and specifying the number of results.

```python
await search_code_examples(ctx, "JWT authentication")
```

```python
await search_code_examples(ctx, "async/await patterns", source_id="docs.example.com")
```

--------------------------------

### Complete Knowledge Graph Workflow Example

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

This Python script demonstrates the full integration process: parsing a GitHub repository into a knowledge graph, analyzing an AI-generated script, validating it against the graph, and generating a report. Ensure Neo4j connection details and environment variables are configured.

```python
# 1. Parse GitHub repository into knowledge graph
extractor = DirectNeo4jExtractor(neo4j_uri, neo4j_user, neo4j_password)
await extractor.initialize()
await extractor.analyze_repository("https://github.com/user/lib.git")

# 2. Analyze AI-generated script
analyzer = AIScriptAnalyzer()
analysis = analyzer.analyze_script("/path/to/generated_script.py")

# 3. Validate against knowledge graph
validator = KnowledgeGraphValidator(neo4j_uri, neo4j_user, neo4j_password)
await validator.initialize()
validation = await validator.validate_script(analysis)

# 4. Generate report
reporter = HallucinationReporter()
report = reporter.generate_comprehensive_report(validation)

# 5. Clean up
await validator.close()
await extractor.close()
```

--------------------------------

### search_code_examples

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Searches for code examples relevant to a query. This tool requires `USE_AGENTIC_RAG=true` to be enabled.

```APIDOC
## search_code_examples

### Description
Searches for code examples relevant to a query. This tool requires `USE_AGENTIC_RAG=true` to be enabled.

### Method
POST

### Endpoint
/tools/search_code_examples

### Parameters
#### Path Parameters
None

#### Query Parameters
* **query** (str) - Required - Search query for code examples.
* **source_id** (str) - Optional - Optional source ID to filter results.
* **match_count** (int) - Optional - Maximum number of results to return. Defaults to 5.

#### Request Body
None (parameters are passed as query parameters)

### Request Example
```python
# Find authentication code examples
examples = await search_code_examples(ctx, "JWT authentication")

# Find examples from a specific source
examples = await search_code_examples(ctx, "async/await patterns", source_id="docs.example.com")
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
- **query** (string) - The original search query.
- **source_filter** (string) - The source filter applied, if any.
- **search_mode** (string) - The search mode used ('hybrid' or 'vector').
- **reranking_applied** (boolean) - Whether reranking was applied.
- **results** (array) - An array of code example results, each containing:
  - **url** (string) - The URL where the code example was found.
  - **code** (string) - The code block.
  - **summary** (string) - An AI-generated summary of the code.
  - **metadata** (object) - Metadata associated with the code example.
  - **source_id** (string) - The ID of the source.
  - **similarity** (float) - The similarity score.
  - **rerank_score** (float, optional) - The rerank score, if applied.
- **count** (integer) - The number of results returned.
- **error** (string, optional) - An error message if the operation failed.

#### Response Example
```json
{
  "success": true,
  "query": "JWT authentication",
  "source_filter": null,
  "search_mode": "vector",
  "reranking_applied": false,
  "results": [
    {
      "url": "https://example.com/code/jwt_auth",
      "code": "def authenticate(token): ...",
      "summary": "Example function to authenticate using JWT.",
      "metadata": {},
      "source_id": "docs.example.com",
      "similarity": 0.92
    }
  ],
  "count": 1
}
```
```

--------------------------------

### Clone Repository (Bash)

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Clone the mcp-crawl4ai-rag repository. This is the first step for both Docker and direct installation methods.

```bash
git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
cd mcp-crawl4ai-rag
```

--------------------------------

### generate_code_example_summary

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Generates a concise summary for a given code example by leveraging the OpenAI API and considering the provided context before and after the code. It falls back to a generic message if the generation process fails.

```APIDOC
## generate_code_example_summary

### Description
Generates a summary for a code example using its context (requires `MODEL_CHOICE` env var).

### Signature
```python
def generate_code_example_summary(code: str, context_before: str, context_after: str) -> str
```

### Parameters
#### Parameters
- **code** (str) - Required - Code content
- **context_before** (str) - Required - Preceding context
- **context_after** (str) - Required - Following context

### Returns
Concise 2-3 sentence summary of the code example

### Description
- Uses OpenAI API with `MODEL_CHOICE` model
- Generates practical summaries based on context
- Falls back to generic message if generation fails

### Example
```python
summary = generate_code_example_summary(
    code="async def authenticate(user, pwd):\n    return await verify(user, pwd)",
    context_before="Here's how to authenticate users:",
    context_after="This pattern ensures secure authentication."
)
# Returns: "Asynchronous authentication function that verifies user credentials..."
```
```

--------------------------------

### Run mcp-crawl4ai-rag Server using Python

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Start the server by running the specified Python script using uv. The server will listen on the configured host and port.

```bash
uv run src/crawl4ai_mcp.py
```

--------------------------------

### Search Code Examples with Vector Similarity

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Use this function to find code examples based on a natural language query. It enhances the query and uses vector similarity to return relevant code blocks with summaries and scores. Filter results by source ID if needed.

```python
def search_code_examples(
    client: Client, 
    query: str, 
    match_count: int = 10, 
    filter_metadata: Optional[Dict[str, Any]] = None,
    source_id: Optional[str] = None
) -> List[Dict[str, Any]]
```

```python
examples = search_code_examples(
    client=supabase,
    query="async authentication patterns",
    match_count=5,
    source_id="docs.example.com"
)
# Returns [{code: "...", summary: "...", similarity: 0.92, ...}, ...]
```

--------------------------------

### add_code_examples_to_supabase

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Adds code examples to the Supabase `code_examples` table in batches, handling deletion of existing entries, embedding creation, and batch insertion with retry logic.

```APIDOC
## add_code_examples_to_supabase

### Description
Adds code examples to the Supabase `code_examples` table in batches.

### Signature
```python
def add_code_examples_to_supabase(
    client: Client,
    urls: List[str],
    chunk_numbers: List[int],
    code_examples: List[str],
    summaries: List[str],
    metadatas: List[Dict[str, Any]],
    batch_size: int = 20
)
```

### Parameters
#### Parameters
- **client** (Client) - Required - Supabase client
- **urls** (List[str]) - Required - Source URLs
- **chunk_numbers** (List[int]) - Required - Code example indices
- **code_examples** (List[str]) - Required - Code contents
- **summaries** (List[str]) - Required - AI-generated summaries
- **metadatas** (List[Dict]) - Required - Metadata for each example
- **batch_size** (int) - Optional - Batch insertion size (default: 20)

### Description
- Deletes existing code examples for same URLs
- Creates embeddings combining code and summary
- Inserts in batches with retry logic
- Falls back to individual insertion if batch fails
```

--------------------------------

### Validate Script Example

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

Demonstrates how to use the KnowledgeGraphValidator to validate a script. Ensure to initialize the validator and obtain an AnalysisResult object first.

```python
validator = KnowledgeGraphValidator("bolt://localhost:7687", "neo4j", "password")
await validator.initialize()

analyzer = AIScriptAnalyzer()
analysis = analyzer.analyze_script("/path/to/script.py")

validation = await validator.validate_script(analysis)
# validation.overall_confidence: 0.87
# validation.hallucinations_detected: [...]
```

--------------------------------

### Search Code Examples - Agentic RAG Disabled Error

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/errors.md

This error indicates that code example extraction is disabled via the `USE_AGENTIC_RAG` configuration. Perform a standard RAG search instead.

```json
{
  "success": false,
  "error": "Code example extraction is disabled. Perform a normal RAG search."
}
```

--------------------------------

### search_code_examples

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

Searches for code examples based on a query. It can filter results by source ID and specify the maximum number of results.

```APIDOC
## search_code_examples

### Description
Searches for code examples based on a query. It can filter results by source ID and specify the maximum number of results. This tool is available only if `USE_AGENTIC_RAG` is set to `true`.

### Method
POST

### Endpoint
/search_code_examples

### Parameters
#### Request Body
- **query** (string) - Required - Search query for code examples
- **source_id** (string) - Optional - Optional source ID to filter results
- **match_count** (integer) - Optional - Maximum number of results to return (default: 5)

### Request Example
```json
{
  "query": "python list comprehension examples",
  "source_id": "github.com/user/repo",
  "match_count": 3
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful
- **query** (string) - The original search query
- **source_filter** (string) - The source filter used, if any
- **search_mode** (string) - The search mode used ('hybrid' or 'vector')
- **reranking_applied** (boolean) - Whether reranking was applied
- **results** (array) - An array of code example results
  - **url** (string) - The URL of the code example
  - **code** (string) - The code snippet
  - **summary** (string) - An AI-generated summary of the code
  - **metadata** (object) - Metadata associated with the code example
  - **source_id** (string) - The ID of the source
  - **similarity** (number) - The similarity score of the result
  - **rerank_score** (number) - The rerank score (optional)
- **count** (integer) - The total number of results returned

#### Response Example
```json
{
  "success": true,
  "query": "python list comprehension examples",
  "source_filter": "github.com/user/repo",
  "search_mode": "hybrid",
  "reranking_applied": true,
  "results": [
    {
      "url": "https://github.com/user/repo/blob/main/example.py",
      "code": "squares = [x**2 for x in range(10)]",
      "summary": "This code generates a list of squares using list comprehension.",
      "metadata": {},
      "source_id": "github.com/user/repo",
      "similarity": 0.92,
      "rerank_score": 0.88
    }
  ],
  "count": 1
}
```
```

--------------------------------

### Enable Agentic RAG for Code Extraction

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Set to 'true' to extract and index code examples separately, enabling dedicated code search capabilities. This increases crawling time and storage. Requires OPENAI_API_KEY and MODEL_CHOICE for summaries.

```bash
USE_AGENTIC_RAG=true
```

--------------------------------

### Get Available Sources Request Schema

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

The request schema for the `get_available_sources` tool, which takes no parameters.

```json
{
  "type": "object",
  "properties": {}
}
```

--------------------------------

### search_code_examples

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Searches for code examples using vector similarity. Enhances query with descriptive terms for better matching and executes an RPC function to return code blocks with summaries and similarity scores.

```APIDOC
## search_code_examples

### Description
Searches for code examples using vector similarity. Enhances query with descriptive terms for better matching and executes an RPC function to return code blocks with summaries and similarity scores.

### Method Signature
```python
def search_code_examples(
    client: Client, 
    query: str, 
    match_count: int = 10, 
    filter_metadata: Optional[Dict[str, Any]] = None,
    source_id: Optional[str] = None
) -> List[Dict[str, Any]]
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters Table
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `client` | `Client` | Yes | — | Supabase client |
| `query` | `str` | Yes | — | Search query |
| `match_count` | `int` | No | 10 | Maximum results |
| `filter_metadata` | `Optional[Dict]` | No | None | Metadata filter |
| `source_id` | `Optional[str]` | No | None | Filter by source ID |

### Returns
List of matching code examples with summaries

### Example
```python
examples = search_code_examples(
    client=supabase,
    query="async authentication patterns",
    match_count=5,
    source_id="docs.example.com"
)
# Returns [{code: "...", summary: "...", similarity: 0.92, ...}, ...]
```
```

--------------------------------

### Get Available Sources Response Structure

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

Details the fields returned by the `get_available_sources` tool, including source ID, summary, word count, and timestamps.

```json
{
  "success": boolean,
  "sources": [
    {
      "source_id": string,
      "summary": string,
      "total_words": integer,
      "created_at": string,
      "updated_at": string
    }
  ],
  "count": integer
}
```

--------------------------------

### Get Available Sources

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Use this function to retrieve a list of all unique sources (domains) that have been crawled and stored in the database. It returns metadata and statistics for each source.

```python
@mcp.tool()
async def get_available_sources(ctx: Context) -> str
```

```python
sources = await get_available_sources(ctx)
# Returns list of crawled domains and their summaries
```

--------------------------------

### Set Neo4j Connection URI

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Configure the Neo4j database connection URI. Examples include local, Docker, or cloud instances. The default is 'bolt://localhost:7687'.

```bash
NEO4J_URI=bolt://localhost:7687
```

```bash
NEO4J_URI=bolt://neo4j:7687
```

```bash
NEO4J_URI=neo4j+s://[instance-id].databases.neo4j.io
```

--------------------------------

### Query Knowledge Graph

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Queries and explores the Neo4j knowledge graph. Requires `USE_KNOWLEDGE_GRAPH=true`. Supports commands for listing repositories, exploring specific repos, getting class details, searching methods, and executing custom Cypher queries.

```python
await query_knowledge_graph(ctx, "repos")
```

```python
await query_knowledge_graph(ctx, "explore pydantic-ai")
```

```python
await query_knowledge_graph(ctx, "class Agent")
```

```python
await query_knowledge_graph(ctx, "method run_stream")
```

```python
await query_knowledge_graph(ctx, "query MATCH (c:Class)-[:HAS_METHOD]->(m:Method) WHERE m.name = 'run' RETURN c.name, m.name LIMIT 5")
```

--------------------------------

### CodeExampleSearchResult Structure

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/types.md

Represents a search result from a code example vector search. It includes details about the code example, its source, and similarity.

```python
{
    "id": str,                # Record ID in Supabase
    "url": str,               # Source URL
    "chunk_number": int,      # Example index
    "content": str,           # Code content
    "summary": str,           # AI-generated summary
    "metadata": dict,         # Example metadata
    "source_id": str,         # Source domain
    "similarity": float       # Similarity score 0.0-1.0
}
```

--------------------------------

### Create and Activate Virtual Environment (Bash)

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Create a new virtual environment using uv and activate it. This isolates project dependencies. The activation command differs for Windows and Mac/Linux.

```bash
uv venv
.venv\Scripts\activate
# on Mac/Linux: source .venv/bin/activate
```

--------------------------------

### Build Docker Image

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Build the Docker image for the application, specifying the tag and the port to be used.

```bash
docker build -t mcp/crawl4ai-rag --build-arg PORT=8051 .
```

--------------------------------

### Supabase Schema Initialization

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

SQL script to initialize the Supabase database schema. Run this in the Supabase SQL Editor.

```sql
-- Run in Supabase SQL Editor
-- This creates:
-- - crawled_pages table (for documents)
-- - code_examples table (for code snippets)
-- - sources table (for metadata)
-- - match_crawled_pages RPC function
-- - match_code_examples RPC function
```

--------------------------------

### Run Docker Container with Environment File

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Run the Docker container, mounting the .env file to provide environment variables and mapping the host port to the container port.

```bash
docker run --env-file .env -p 8051:8051 mcp/crawl4ai-rag
```

--------------------------------

### get_supabase_client

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/utils.md

Initializes and returns a configured Supabase client instance using environment variables.

```APIDOC
## get_supabase_client

### Description
Creates and returns a Supabase client instance. Initializes Supabase client using environment variables. Must be called after setting `SUPABASE_URL` and `SUPABASE_SERVICE_KEY`.

### Returns
- `Client`: Configured Supabase `Client` instance

### Raises
- `ValueError`: if `SUPABASE_URL` or `SUPABASE_SERVICE_KEY` environment variables are not set

### Example
```python
client = get_supabase_client()
# Use client to interact with Supabase tables
```
```

--------------------------------

### Get Available Sources Response Schema

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

Defines the JSON structure for the `get_available_sources` tool response, containing a list of crawled sources.

```json
{
  "type": "string",
  "description": "JSON string with sources list"
}
```

--------------------------------

### FastMCP Server Initialization

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Initializes the FastMCP server instance, configured with necessary tools and lifespan management for RAG and web crawling.

```APIDOC
## mcp Server Initialization

### Description
FastMCP server instance configured with all tools and lifespan management.

### Initialization Parameters
- **project_name** (str) - Required - The name of the project, e.g., "mcp-crawl4ai-rag".
- **description** (str) - Optional - A description for the MCP server, e.g., "MCP server for RAG and web crawling with Crawl4AI".
- **lifespan** - Required - The lifespan management function, e.g., `crawl4ai_lifespan`.
- **host** (str) - Optional - The server host address (defaults to environment variable `HOST` or "0.0.0.0").
- **port** (int) - Optional - The server port number (defaults to environment variable `PORT` or 8051).
```

--------------------------------

### SSE Configuration for Windsurf Users

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

For Windsurf users, use 'serverUrl' instead of 'url' in the SSE configuration.

```json
{
  "mcpServers": {
    "crawl4ai-rag": {
      "transport": "sse",
      "serverUrl": "http://localhost:8051/sse"
    }
  }
}
```

--------------------------------

### Set Server Host

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Configure the server's binding address. Defaults to '0.0.0.0' for all interfaces.

```bash
HOST=localhost
```

```bash
HOST=0.0.0.0
```

--------------------------------

### Clone Local AI Package (Bash)

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Clone the Local AI Package repository, which includes Neo4j and other AI services. This is recommended for setting up Neo4j locally.

```bash
git clone https://github.com/coleam00/local-ai-packaged.git
cd local-ai-packaged
```

--------------------------------

### SSE Integration Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

Configure the crawl4ai-rag server to use Server-Sent Events (SSE) for transport. Ensure the server is started with TRANSPORT=sse and listens on the /sse endpoint.

```json
{
  "mcpServers": {
    "crawl4ai-rag": {
      "transport": "sse",
      "url": "http://localhost:8051/sse"
    }
  }
}
```

--------------------------------

### crawl_single_page

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Crawls a single web page, chunks its markdown content, stores the chunks with embeddings in Supabase, and extracts code examples if configured. It also updates source information in the database.

```APIDOC
## crawl_single_page

### Description
Crawls a single web page and stores its content in Supabase. This function chunks the markdown content, stores these chunks with embeddings, and optionally extracts code examples.

### Method
`@mcp.tool()`

### Parameters
#### Path Parameters
- **ctx** (Context) - Required - MCP server context
- **url** (str) - Required - URL of the web page to crawl

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the crawl was successful.
- **url** (string) - The URL that was crawled.
- **chunks_stored** (integer) - The number of content chunks stored.
- **code_examples_stored** (integer) - The number of code examples stored.
- **content_length** (integer) - The total length of the content in characters.
- **total_word_count** (integer) - The total word count of the content.
- **source_id** (string) - The unique identifier for the source.
- **links_count** (object) - An object containing counts of internal and external links.
  - **internal** (integer) - Count of internal links.
  - **external** (integer) - Count of external links.

#### Error Response
- **success** (boolean) - Always false on error.
- **error** (string) - Description of the error that occurred.

### Request Example
```python
result = await crawl_single_page(ctx, "https://docs.example.com/guide")
# Returns JSON with crawl statistics
```
```

--------------------------------

### Stdio Configuration for MCP Clients

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Configure the server for MCP clients using the Stdio transport. This includes specifying the command, arguments, and environment variables.

```json
{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "python",
      "args": ["path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}
```

--------------------------------

### MCP Server and API Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Define server host, port, transport, and API keys for OpenAI and Supabase. Also configure RAG strategy options and Neo4j credentials if using a knowledge graph.

```env
# MCP Server Configuration
HOST=0.0.0.0
PORT=8051
TRANSPORT=sse

# OpenAI API Configuration
OPENAI_API_KEY=sk-proj-...your-key-here...
MODEL_CHOICE=gpt-4-mini

# Supabase Configuration
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

# RAG Strategy Options
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=false
USE_RERANKING=true
USE_KNOWLEDGE_GRAPH=false

# Neo4j Configuration (for knowledge graph)
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_neo4j_password
```

--------------------------------

### crawl_recursive_internal_links

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Recursively crawls internal links from specified starting URLs up to a defined depth and concurrency limit. It returns a list of dictionaries, each containing 'url' and 'markdown' fields for all crawled pages.

```APIDOC
## crawl_recursive_internal_links

### Description
Recursively crawls internal links from starting URLs up to a specified depth.

### Parameters
- **crawler** (AsyncWebCrawler) - Required - The asynchronous web crawler instance.
- **start_urls** (List[str]) - Required - A list of URLs to start the recursive crawling from.
- **max_depth** (int) - Optional - The maximum depth for recursive crawling (default: 3).
- **max_concurrent** (int) - Optional - The maximum number of concurrent crawling tasks (default: 10).

### Returns
- List[Dict[str, Any]] - A list of dictionaries with 'url' and 'markdown' fields for all crawled pages.
```

--------------------------------

### Docker with Stdio Configuration for MCP Clients

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Integrate the server with MCP clients using Docker and Stdio transport. This configuration maps necessary environment variables for the Docker container.

```json
{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "docker",
      "args": ["run", "--rm", "-i", 
               "-e", "TRANSPORT", 
               "-e", "OPENAI_API_KEY", 
               "-e", "SUPABASE_URL", 
               "-e", "SUPABASE_SERVICE_KEY",
               "-e", "USE_KNOWLEDGE_GRAPH",
               "-e", "NEO4J_URI",
               "-e", "NEO4J_USER",
               "-e", "NEO4J_PASSWORD",
               "mcp/crawl4ai"],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}
```

--------------------------------

### crawl_single_page

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

Crawl and index a single web page. This tool is always available and requires a URL as input. It returns crawl results including stored chunks, code examples, and content statistics.

```APIDOC
## crawl_single_page

### Description
Crawl and index a single web page.

### Method
POST

### Endpoint
/crawl_single_page

### Parameters
#### Request Body
- **url** (string) - Required - URL of the web page to crawl

### Request Example
{
  "url": "https://example.com"
}

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the crawl was successful
- **url** (string) - The URL that was crawled
- **chunks_stored** (integer) - Number of content chunks stored
- **code_examples_stored** (integer) - Number of code examples stored
- **content_length** (integer) - Total length of the content in characters
- **total_word_count** (integer) - Total word count of the content
- **source_id** (string) - Identifier for the crawled source
- **links_count** (object) - Count of internal and external links found
  - **internal** (integer)
  - **external** (integer)

#### Response Example
{
  "success": true,
  "url": "https://example.com",
  "chunks_stored": 15,
  "code_examples_stored": 2,
  "content_length": 7500,
  "total_word_count": 1200,
  "source_id": "example.com",
  "links_count": {
    "internal": 5,
    "external": 2
  }
}
```

--------------------------------

### Initialize FastMCP Server Instance

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Initializes a FastMCP server instance named 'mcp-crawl4ai-rag'. This instance is configured with a description, lifespan management, and specific host and port settings, defaulting to '0.0.0.0' and '8051' respectively, which can be overridden by environment variables.

```python
mcp = FastMCP(
    "mcp-crawl4ai-rag",
    description="MCP server for RAG and web crawling with Crawl4AI",
    lifespan=crawl4ai_lifespan,
    host=os.getenv("HOST", "0.0.0.0"),
    port=os.getenv("PORT", "8051")
)
```

--------------------------------

### Set Supabase URL

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Provide your Supabase project URL. This is required for Supabase integration.

```bash
SUPABASE_URL=https://myproject.supabase.co
```

--------------------------------

### Crawl Single Web Page

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Use this function to crawl a single web page. It stores the content in Supabase, chunks the markdown, and extracts code examples if agentic RAG is enabled. It returns statistics about the crawl operation.

```python
@mcp.tool()
async def crawl_single_page(ctx: Context, url: str) -> str
```

```python
result = await crawl_single_page(ctx, "https://docs.example.com/guide")
# Returns JSON with crawl statistics
```

--------------------------------

### initialize

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

Initializes the Neo4j connection for the validator. This method must be called before any validation operations.

```APIDOC
## KnowledgeGraphValidator.initialize

### Description
Initializes Neo4j connection. Must be called before using validator. Creates async driver and connects to Neo4j.

### Method
initialize

### Endpoint
async def initialize(self) -> None
```

--------------------------------

### crawl4ai_lifespan Async Context Manager

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Manages the initialization and cleanup of the Crawl4AI MCP Server's resources. It sets up the web crawler, Supabase client, and optionally loads reranking and knowledge graph components based on configuration. Raises ValueError if Supabase credentials are not provided.

```python
@asynccontextmanager
async def crawl4ai_lifespan(server: FastMCP) -> AsyncIterator[Crawl4AIContext]:
    # Initialize AsyncWebCrawler
    crawler = AsyncWebCrawler(
        headless=True,  # Or False, depending on configuration
        # Add other crawler configurations if necessary
    )

    # Initialize Supabase client
    supabase_url = os.environ.get("SUPABASE_URL")
    supabase_key = os.environ.get("SUPABASE_KEY")
    if not supabase_url or not supabase_key:
        raise ValueError("Supabase URL and Key must be set in environment variables.")
    supabase_client: Client = Client(supabase_url, supabase_key)

    # Initialize optional components
    reranking_model: Optional[CrossEncoder] = None
    if os.environ.get("USE_RERANKING", "false").lower() == "true":
        # Load reranking model (e.g., from a local path or Hugging Face)
        # reranking_model = CrossEncoder("path/to/your/reranking/model")
        pass # Placeholder for actual model loading

    knowledge_validator: Optional[Any] = None
    repo_extractor: Optional[Any] = None
    if os.environ.get("USE_KNOWLEDGE_GRAPH", "false").lower() == "true":
        # Initialize KnowledgeGraphValidator and DirectNeo4jExtractor
        # neo4j_uri = os.environ.get("NEO4J_URI")
        # neo4j_user = os.environ.get("NEO4J_USER")
        # neo4j_password = os.environ.get("NEO4J_PASSWORD")
        # if neo4j_uri and neo4j_user and neo4j_password:
        #     knowledge_validator = KnowledgeGraphValidator(neo4j_uri, neo4j_user, neo4j_password)
        #     repo_extractor = DirectNeo4jExtractor(neo4j_uri, neo4j_user, neo4j_password)
        pass # Placeholder for actual initialization

    context = Crawl4AIContext(
        crawler=crawler,
        supabase_client=supabase_client,
        reranking_model=reranking_model,
        knowledge_validator=knowledge_validator,
        repo_extractor=repo_extractor,
    )

    try:
        yield context
    finally:
        # Clean up resources
        await crawler.close()
        # Close Supabase client if necessary (depends on Supabase client library)
        # Close Neo4j connections if necessary
        pass
```

--------------------------------

### Check Neo4j Connectivity

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/errors.md

Ensure Neo4j is running and the URI is correctly formatted. Verify port accessibility and check for firewall issues to resolve 'Cannot connect to Neo4j' errors.

```bash
# Start Neo4j: neo4j start or Docker container
# Verify NEO4J_URI format: bolt://localhost:7687
# Check port is accessible
# Verify no firewall blocking connection
```

--------------------------------

### Define Async Recursive Internal Links Crawl Function

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/crawl4ai-mcp-server.md

Defines an asynchronous function to recursively crawl internal links starting from specified URLs up to a defined maximum depth and concurrency. It returns a list of dictionaries containing 'url' and 'markdown' fields for all successfully crawled pages.

```python
async def crawl_recursive_internal_links(
    crawler: AsyncWebCrawler,
    start_urls: List[str],
    max_depth: int = 3,
    max_concurrent: int = 10
) -> List[Dict[str, Any]]
```

--------------------------------

### Standard RAG Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

This recommended profile offers a balance of performance and accuracy for production environments. It enables hybrid search and reranking for improved result ordering.

```env
# Server
HOST=0.0.0.0
PORT=8051
TRANSPORT=sse

# OpenAI
OPENAI_API_KEY=sk-...
MODEL_CHOICE=gpt-4-mini

# Supabase
SUPABASE_URL=https://myproject.supabase.co
SUPABASE_SERVICE_KEY=eyJhbGci...

# RAG Strategies
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=true
USE_AGENTIC_RAG=false
USE_RERANKING=true
USE_KNOWLEDGE_GRAPH=false
```

--------------------------------

### Stdio Integration Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

Configure the crawl4ai-rag server to use standard input/output (stdio) for communication. The client acts as the parent process. Ensure necessary environment variables are set.

```json
{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "python",
      "args": ["path/to/crawl4ai_mcp.py"],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "...",
        "SUPABASE_URL": "...",
        "SUPABASE_SERVICE_KEY": "..."
      }
    }
  }
}
```

--------------------------------

### AIScriptAnalyzer Class Initialization

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

Initializes the AIScriptAnalyzer. No parameters are required for initialization.

```python
class AIScriptAnalyzer:
    def __init__(self):
        pass
```

--------------------------------

### Set Supabase Credentials

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/errors.md

Ensure Supabase URL and service key are set in the environment. This is required on server startup or before the first database operation.

```env
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_KEY=your-service-key
```

--------------------------------

### KnowledgeGraphValidator Initialize Method

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

Establishes the asynchronous connection to Neo4j. This must be called before any validation operations.

```python
async def initialize(self) -> None
```

--------------------------------

### Add SSE Server using Claude Code CLI

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/README.md

Use this command to add the SSE server configuration via the Claude Code CLI. Replace 'crawl4ai-rag' with your server name if different.

```bash
claude mcp add-json crawl4ai-rag '{"type":"http","url":"http://localhost:8051/sse"}' --scope user
```

--------------------------------

### Enable Contextual Embeddings

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Set to 'true' to enable contextual embeddings for improved semantic understanding, which enhances retrieval accuracy but increases indexing time and cost. Requires OPENAI_API_KEY and MODEL_CHOICE.

```bash
USE_CONTEXTUAL_EMBEDDINGS=true
```

--------------------------------

### Set LLM Model Choice

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Specify the language model for contextual embeddings and summaries. Recommended options include 'gpt-4-mini' or 'gpt-3.5-turbo'.

```bash
MODEL_CHOICE=gpt-4-mini
```

--------------------------------

### Fast Indexing Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Optimize for fast indexing by disabling contextual embeddings and agentic RAG, and using a faster LLM like gpt-3.5-turbo.

```env
USE_CONTEXTUAL_EMBEDDINGS=false
USE_AGENTIC_RAG=false
MODEL_CHOICE=gpt-3.5-turbo  # Faster LLM
```

--------------------------------

### Monitor Supabase

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/errors.md

This command checks the Supabase service by querying the 'crawled_pages' table.

```bash
# Supabase
curl https://your-project.supabase.co/rest/v1/crawled_pages?limit=1
```

--------------------------------

### DirectNeo4jExtractor

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/api-reference/knowledge-graph.md

Initializes the extractor with Neo4j credentials and provides methods to manage the connection and analyze repositories.

```APIDOC
## DirectNeo4jExtractor

### Description
Initializes the extractor with Neo4j credentials and provides methods to manage the connection and analyze repositories.

### Methods

#### `__init__`

Initializes extractor with Neo4j credentials.

**Parameters**:
- `neo4j_uri` (str) - Required - Neo4j connection URI
- `neo4j_user` (str) - Required - Neo4j username
- `neo4j_password` (str) - Required - Neo4j password

#### `initialize`

Initializes Neo4j connection.

**Description**: Creates async driver and verifies connection.

#### `close`

Closes Neo4j connection.

#### `analyze_repository`

Clones and analyzes a GitHub repository, storing results in Neo4j.

**Parameters**:
- `repo_url` (str) - Required - GitHub repository URL ending with .git

**Description**:
- Clones repository to temporary directory
- Analyzes all Python files
- Extracts classes, methods, functions, imports
- Creates Neo4j nodes and relationships
- Automatically detects module names

**Example**:
```python
extractor = DirectNeo4jExtractor("bolt://localhost:7687", "neo4j", "password")
await extractor.initialize()

await extractor.analyze_repository("https://github.com/pydantic/pydantic-ai.git")
# Repository now available for validation
```
```

--------------------------------

### Set Neo4j Username

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Specify the username for connecting to the Neo4j database. The default is 'neo4j'.

```bash
NEO4J_USER=neo4j
```

--------------------------------

### Docker Integration Configuration

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/endpoints.md

Configure the crawl4ai-rag server to run as a Docker container using stdio for communication. Environment variables are passed to the container. Ensure the Docker image is available.

```json
{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "-e", "TRANSPORT", "-e", "OPENAI_API_KEY", "-e", "SUPABASE_URL", "-e", "SUPABASE_SERVICE_KEY", "mcp/crawl4ai-rag"],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "...",
        "SUPABASE_URL": "...",
        "SUPABASE_SERVICE_KEY": "..."
      }
    }
  }
}
```

--------------------------------

### Enable Knowledge Graph for Hallucination Detection

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/configuration.md

Set to 'true' to enable Neo4j knowledge graph integration for hallucination detection. This requires Neo4j infrastructure and the corresponding connection details (URI, User, Password).

```bash
USE_KNOWLEDGE_GRAPH=true
```

--------------------------------

### Monitor OpenAI API

Source: https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/_autodocs/errors.md

This command checks the connectivity to the OpenAI API by listing available models.

```bash
# OpenAI
curl https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"
```