### Development Setup and Testing

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

This bash script outlines the steps for setting up the development environment for the project. It covers forking and cloning the repository, creating and activating a virtual environment, installing dependencies, running tests using pytest, and formatting code with black. This is essential for developers contributing to the project.

```bash
# Fork and clone
git clone https://github.com/yourusername/intelligence-agent.git
cd intelligence-agent

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dev dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt  # pytest, black, etc.

# Run tests
pytest tests/

# Format code
black .
```

--------------------------------

### Install Project Dependencies and Modal CLI

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Installs the necessary Python packages for the project and the Modal CLI for deployment. Requires Python 3.10+.

```bash
git clone https://github.com/yourusername/intelligence-agent.git
cd intelligence-agent
pip install -r requirements.txt
pip install modal
modal token new
```

--------------------------------

### AI Plan JSON Example

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

An example of a JSON-formatted AI plan generated for investigating APT29 Cozy Bear activities. It outlines sequential steps involving semantic search, document content retrieval, and summarization.

```json
[
  {
    "step": 1,
    "tool": "semantic_search",
    "parameters": {"query": "APT29 Cozy Bear TTPs 2024", "k": 5}
  },
  {
    "step": 2,
    "tool": "get_document_content",
    "parameters": {"doc_id": "RESULT_FROM_STEP_1.results[0].doc_id"}
  },
  {
    "step": 3,
    "tool": "summarize_with_maverick",
    "parameters": {"text": "RESULT_FROM_STEP_2.content"}
  }
]
```

--------------------------------

### Example Mission Execution via Telegram Bot

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Demonstrates how to send a mission to the Telegram bot for execution. The bot automatically processes the request and returns a report.

```text
Investigate APT29 Cozy Bear activities in 2024
```

--------------------------------

### Report Markdown Example

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

An example of a generated report in Markdown format, summarizing findings related to APT29 Cozy Bear's recent activities, including an executive summary, key findings, and detailed analysis.

```markdown
## Executive Summary

APT29 (Cozy Bear) has demonstrated increased sophistication in 2024,
with focus on supply chain compromises and cloud infrastructure targeting...

## Key Findings

- Enhanced use of legitimate cloud services for C2
- Shift to stealthier, longer-term operations
- Targeting of diplomatic and think tank entities

## Detailed Analysis

Recent campaigns show APT29 leveraging zero-day vulnerabilities...

## Conclusions

Organizations should implement enhanced monitoring...
```

--------------------------------

### Telegram Bot Commands

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Lists the available commands for interacting with the Telegram bot, including starting the bot, getting help, executing missions, and checking status.

```markdown
| Command | Description |
|---------|-------------|
| `/start` | Welcome message & help |
| `/help` | Detailed usage guide |
| `/mission <text>` | Execute investigation |
| `/status` | System health check |
| `/stats` | Your usage statistics |
```

--------------------------------

### GET /metrics

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Returns detailed system metrics including version information.

```APIDOC
## GET /metrics

### Description
Returns detailed system metrics including version information.

### Method
GET

### Endpoint
/metrics

### Parameters
None

### Request Example
```bash
curl "https://your-app.modal.run/metrics"
```

### Response
#### Success Response (200)
- **timestamp** (number) - The timestamp when the metrics were collected.
- **system** (object) - System-related metrics.
  - **status** (string) - The overall health status of the system.
  - **uptime** (number) - The system uptime in seconds.
  - **documents** (integer) - The number of documents currently stored.
  - **cache_size** (integer) - The current size of the cache in bytes.
- **version** (string) - The version of the Intelligence Agent System.

#### Response Example
```json
{
  "timestamp": 1699900000.0,
  "system": {
    "status": "healthy",
    "uptime": 3600.5,
    "documents": 150,
    "cache_size": 1024000
  },
  "version": "2.0.7-final"
}
```
```

--------------------------------

### Deploy and Test Intelligence Agent System with Modal CLI (Bash)

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

This bash script outlines the steps to deploy the Intelligence Agent System using the Modal CLI. It covers installing the Modal CLI, authenticating with Modal, creating a secret for Cerebras API keys, deploying the application, and testing the deployed endpoint. The output of the deployment command provides the URL for the running application.

```bash
# Install Modal CLI
pip install modal

# Authenticate with Modal
modal token new

# Create Cerebras API secret
modal secret create cerebras-api-key CEREBRAS_API_KEY="sk-your-cerebras-key-here"

# Deploy to Modal
modal deploy app.py
# Output: https://your-username--intelligence-agent-system-fixed-fastapi-app.modal.run

# Test deployment
curl https://your-username--intelligence-agent-system-fixed-fastapi-app.modal.run/health

# Run locally for development
python app.py
# Server starts at http://localhost:8000
```

--------------------------------

### AI Architect Tool Chaining Example (JSON)

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

This JSON object demonstrates how the AI Architect chains tools together to perform a complex task. It outlines a plan involving semantic search, document content retrieval, and summarization, showing parameter passing between steps.

```json
[
  {
    "tool_name": "semantic_search",
    "parameters": {"query": "LockBit ransomware", "k": 3},
    "description": "Search for LockBit documents"
  },
  {
    "tool_name": "get_document_content",
    "parameters": {"doc_id": "RESULT_FROM_STEP_1.results[0].doc_id"},
    "description": "Get full content of top result"
  },
  {
    "tool_name": "summarize_with_maverick",
    "parameters": {"text": "RESULT_FROM_STEP_2.content"},
    "description": "Summarize the document"
  }
]
```

--------------------------------

### Get System Metrics (API)

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Retrieves system metrics by sending a GET request to the /metrics endpoint. Returns timestamp, system status, uptime, document count, cache size, and API version.

```bash
curl https://your-url.modal.run/metrics
```

--------------------------------

### Modal Deployment Dockerfile

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

A Dockerfile for deploying the intelligence agent application. It sets up a Python environment, installs dependencies, copies application files, and configures the command to run the application using uvicorn.

```dockerfile
FROM python:3.10-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

ENV CEREBRAS_API_KEY=your-key

CMD ["uvicorn", "app:web_app", "--host", "0.0.0.0", "--port", "8000"]
```

--------------------------------

### Telegram Bot Interaction Example

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Demonstrates how to interact with the Telegram bot for intelligence investigations. Users send a natural language query to the bot, which then initiates an investigation process.

```text
Investigate LockBit ransomware activities
```

--------------------------------

### Telegram Bot Configuration and Commands

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Configuration and usage examples for the Telegram bot integration. Enables mobile access to the intelligence agent with rate limiting and inline keyboards. Requires setting Telegram token and API base URL.

```python
# telegram_bot.py configuration
TELEGRAM_TOKEN = os.getenv("TELEGRAM_BOT_TOKEN", "YOUR_BOT_TOKEN")
API_BASE_URL = os.getenv("API_BASE_URL", "https://your-app.modal.run")
RATE_LIMIT_SECONDS = 60  # 1 mission per minute per user
ADMIN_USER_IDS = [123456789]  # Admin Telegram user IDs

# Start the bot
python telegram_bot.py

# Available bot commands:
# /start - Show welcome message and help
# /help - Detailed usage guide with examples
# /mission <description> - Execute intelligence mission
# /status - Check system health
# /stats - View your usage statistics

# Direct message example (no command needed):
# User: Investigate APT29 Cozy Bear activities in 2024
# Bot: Executing Mission...
# Bot: Mission Complete
#      ## Executive Summary
#      APT29 has demonstrated increased sophistication...
#      ## Key Findings
#      - Enhanced use of legitimate cloud services for C2
#      - Shift to stealthier, longer-term operations
#      ...
#      Completed in 5.6s | 3/3 steps successful

# Rate limiting response:
# User: (sends another mission within 60 seconds)
# Bot: Please wait 45s before submitting another mission.

```

--------------------------------

### Get Document Content API

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Retrieves the full content of a specific document using its document ID. This is useful for getting detailed information after a semantic search.

```APIDOC
## GET /get_document_content

### Description
Retrieves the full content of a specific document by its ID from the shelve database.

### Method
GET

### Endpoint
/get_document_content

#### Query Parameters
- **doc_id** (integer) - Required - The Document ID obtained from search results.

### Request Body
This endpoint does not require a request body. The `doc_id` is passed as a query parameter.

### Request Example
```python
system = IntelligenceSystem()

# Get full document content for document ID 0
result = system.get_document_content(doc_id=0)
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the status of the operation ('success' or 'error').
- **doc_id** (integer) - The unique identifier for the document.
- **title** (string) - The title of the document.
- **url** (string) - The URL of the document.
- **content** (string) - The full text content of the document.
- **length** (integer) - The character length of the document content.

#### Response Example
```json
{
  "status": "success",
  "doc_id": 0,
  "title": "LockBit Ransomware Overview",
  "url": "https://example.com/lockbit",
  "content": "LockBit is a ransomware-as-a-service (RaaS) operation that emerged in September 2019. It is known for its automated encryption processes and affiliate model. LockBit 3.0, released in 2022, introduced enhanced capabilities including the world's first ransomware bug bounty program. The group has targeted organizations across various sectors globally, employing double extortion tactics.",
  "length": 420
}
```
```

--------------------------------

### Python Semantic Search Tool

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Defines the signature and provides an example usage for the `semantic_search` tool within the Intelligence Agent System. This tool allows searching documents based on semantic understanding.

```python
def semantic_search(query: str, k: int = 3) -> Dict[str, Any]:
    """Search for relevant documents using semantic understanding"""
    # Implementation details omitted
    pass

# Example Usage:
# Find documents about ransomware
result = agent.semantic_search("ransomware attack techniques", k=5)
```

--------------------------------

### Test Health Endpoint via cURL

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Sends a GET request to the health endpoint of the deployed Modal application to verify its status. Replace 'your-url.modal.run' with the actual deployment URL.

```bash
curl https://your-url.modal.run/health
```

--------------------------------

### GET /health

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Returns system health status including document count, cache size, uptime, and Cerebras API mode.

```APIDOC
## GET /health

### Description
Returns system health status including document count, cache size, uptime, and Cerebras API mode.

### Method
GET

### Endpoint
/health

### Parameters
None

### Request Example
```bash
curl "https://your-app.modal.run/health"
```

### Response
#### Success Response (200)
- **status** (string) - The overall health status of the system (e.g., "healthy").
- **uptime** (number) - The system uptime in seconds.
- **documents** (integer) - The number of documents currently stored.
- **cache_size** (integer) - The current size of the cache in bytes.
- **vector_dimension** (integer) - The dimension of the vectors used in the database.
- **model** (string) - The name of the sentence transformer model used for embeddings.
- **cerebras_mode** (string) - The current mode of the Cerebras API.
- **api_url** (string) - The URL of the Cerebras API endpoint.
- **python_version** (string) - The Python version used by the system.

#### Response Example
```json
{
  "status": "healthy",
  "uptime": 3600.5,
  "documents": 150,
  "cache_size": 1024000,
  "vector_dimension": 384,
  "model": "all-MiniLM-L6-v2",
  "cerebras_mode": "SDK",
  "api_url": "https://api.cerebras.ai/v1/chat/completions",
  "python_version": "3.11.0"
}
```
```

--------------------------------

### Execute Mission via REST API using cURL

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Sends a POST request to the /api/execute_mission endpoint to start an intelligence mission. The mission details are provided in JSON format in the request body.

```bash
curl -X POST https://your-url.modal.run/api/execute_mission \
  -H "Content-Type: application/json" \
  -d '{"mission": "Search for information about ransomware"}'
```

--------------------------------

### Add Sample Documents to the Agent

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Runs a Python script to add sample documents to the intelligence agent. Requires the API URL of the deployed agent.

```bash
python add_documents.py \
  --api-url https://your-url.modal.run \
  --sample
```

--------------------------------

### Semantic Search Implementation (Python)

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Demonstrates the steps involved in semantic search using sentence transformers and FAISS. Includes text encoding, vector similarity search, result ranking, and metadata retrieval.

```python
# 1. Text Encoding
user_query = "ransomware attacks"
query_vector = sentence_transformer.encode(user_query)
# Output: 384-dimensional vector

# 2. Vector Similarity Search
distances, indices = faiss_index.search(query_vector, k=3)
# Uses L2 distance for similarity

# 3. Result Ranking
scores = [1 - distance for distance in distances[0]]
# Convert distance to similarity score (0-1)

# 4. Metadata Retrieval
results = []
for idx, score in zip(indices[0], scores):
    doc_metadata = shelve_db[str(idx)]
    results.append({
        "doc_id": idx,
        "score": score,
        "title": doc_metadata['title'],
        ...
    })
```

--------------------------------

### Add Documents Endpoint

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Adds sample documents to the system.

```APIDOC
## POST /add_documents

### Description
Adds sample documents to the system. This endpoint is typically used for initial setup or testing.

### Method
POST

### Endpoint
/add_documents

### Parameters
#### Path Parameters
None

#### Query Parameters
- **api-url** (string) - Required - The base URL of the API.
- **sample** (boolean) - Optional - If true, adds sample documents.

### Request Example
```bash
python add_documents.py \
  --api-url https://your-url.modal.run \
  --sample
```

### Response
#### Success Response (200)
- **message** (string) - A confirmation message indicating that documents were added.

#### Response Example
```json
{
  "message": "Sample documents added successfully."
}
```
```

--------------------------------

### AI Planning with System Prompt

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Defines the system prompt for the 'Architect' AI planner, enabling it to use available tools like semantic search, document retrieval, and summarization for investigation planning. It utilizes few-shot learning with a JSON output format for the plan.

```python
system_prompt = """
You are "Architect", an AI planner for investigations.

Available tools:
1. semantic_search(query, k) - Find relevant documents
2. get_document_content(doc_id) - Get full document
3. summarize_with_maverick(text) - Summarize text

Output a JSON plan with steps to achieve the mission.
Use placeholders for dependent parameters.

Example:
Mission: "Research LockBit ransomware"

Plan:
[
  {
    "tool_name": "semantic_search",
    "parameters": {"query": "LockBit ransomware", "k": 3},
    "description": "Find relevant documents"
  },
  {
    "tool_name": "get_document_content",
    "parameters": {"doc_id": "RESULT_FROM_STEP_1.results[0].doc_id"},
    "description": "Get top result content"
  }
]
"""
```

--------------------------------

### Deploy Intelligence Agent Application

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Deploys the FastAPI application to Modal. After deployment, a URL will be provided for accessing the service.

```bash
modal deploy app.py
```

--------------------------------

### Configure and Deploy Intelligence Agent System to Modal (Python)

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

This Python code configures the Modal application for the Intelligence Agent System. It sets up the Modal app, defines a persistent volume for data, specifies a secret for API keys, and defines the Docker image with necessary Python dependencies. The code also includes a function to expose a FastAPI application, configured with resources like CPU, memory, and timeouts for serverless deployment.

```python
import modal

app = modal.App(name="intelligence-agent-system-fixed")
volume = modal.Volume.from_name("intelligence-data-vol", create_if_missing=True)
cerebras_secret = modal.Secret.from_name("cerebras-api-key")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install(
        "fastapi", "uvicorn[standard]", "pydantic", "numpy", "torch",
        "transformers", "sentence-transformers", "faiss-cpu", "requests"
    )
    .run_commands("pip install cerebras-cloud-sdk --no-deps --force-reinstall || echo 'SDK install failed'")
)

@app.function(
    image=image,
    secrets=[cerebras_secret],
    volumes={"/cache": volume},
    cpu=2.0,
    memory=4096,
    timeout=900,
    keep_warm=1,
    container_idle_timeout=300
)
@modal.asgi_app()
def fastapi_app():
    return web_app
```

--------------------------------

### Bulk Document Import CLI (add_documents.py)

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Command-line tool for bulk importing documents into the knowledge base. Supports sample data, JSON file import, and custom delays.

```APIDOC
## CLI Commands for add_documents.py

### Description
This section outlines the commands and options for the `add_documents.py` script, used for bulk importing documents.

### Usage

- **Import sample documents:**
  ```bash
  python add_documents.py --api-url <your_api_url> --sample
  ```

- **Import documents from a JSON file:**
  ```bash
  python add_documents.py --api-url <your_api_url> --json <path_to_your_json_file>
  ```

- **Create a sample JSON template:**
  ```bash
  python add_documents.py --create-sample
  ```

- **Import with custom delay for rate limiting:**
  ```bash
  python add_documents.py --api-url <your_api_url> --json <path_to_your_json_file> --delay <seconds>
  ```

### Parameters

- **`--api-url`**: (string) - Required - The base URL of the Intelligence Agent API.
- **`--sample`**: (flag) - Optional - Imports predefined sample documents.
- **`--json`**: (string) - Optional - Path to a JSON file containing documents to import. The JSON file should be an array of objects, each with `title`, `url`, and `text` fields.
- **`--create-sample`**: (flag) - Optional - Creates a sample `documents.json` file.
- **`--delay`**: (float) - Optional - Custom delay in seconds between requests to manage rate limiting.

### JSON File Format Example
```json
[
  {
    "title": "Document Title",
    "url": "https://source-url.com",
    "text": "Full document content here (minimum 50 characters)..."
  },
  {
    "title": "Another Document",
    "url": "https://another-source.com",
    "text": "Another document's full text content..."
  }
]
```

### Output Example
```
2024-01-15 10:30:00 - INFO - Using API: https://your-app.modal.run
2024-01-15 10:30:00 - INFO - Starting bulk import of 5 documents...
2024-01-15 10:30:00 - INFO - [1/5] Processing: LockBit Ransomware Overview
2024-01-15 10:30:01 - INFO - Added 'LockBit Ransomware Overview' (ID: 0)
...
============================================================
IMPORT SUMMARY
============================================================
Total Documents: 5
Successfully Added: 5
Failed: 0
Success Rate: 100.0%
============================================================
```
```

--------------------------------

### Bulk Document Import CLI Usage

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Command-line tool for bulk importing documents into the knowledge base. Supports importing sample documents, from a JSON file, and creating a sample JSON template. Allows custom delay for rate limiting.

```bash
# Add predefined sample cybersecurity documents
python add_documents.py --api-url https://your-app.modal.run --sample

# Import documents from a JSON file
python add_documents.py --api-url https://your-app.modal.run --json documents.json

# Create a sample JSON template for custom documents
python add_documents.py --create-sample

# Import with custom delay between requests (for rate limiting)
python add_documents.py --api-url https://your-app.modal.run --json documents.json --delay 2.0

# Expected JSON format for documents.json:
# [
#   {
#     "title": "Document Title",
#     "url": "https://source-url.com",
#     "text": "Full document content here (minimum 50 characters)..."
#   },
#   {
#     "title": "Another Document",
#     "url": "https://another-source.com",
#     "text": "Another document's full text content..."
#   }
# ]

# Output:
# 2024-01-15 10:30:00 - INFO - Using API: https://your-app.modal.run
# 2024-01-15 10:30:00 - INFO - Starting bulk import of 5 documents...
# 2024-01-15 10:30:00 - INFO - [1/5] Processing: LockBit Ransomware Overview
# 2024-01-15 10:30:01 - INFO - Added 'LockBit Ransomware Overview' (ID: 0)
# ...
# ============================================================
# IMPORT SUMMARY
# ============================================================
# Total Documents: 5
# Successfully Added: 5
# Failed: 0
# Success Rate: 100.0%
# ============================================================

```

--------------------------------

### Python SDK for Deepweb Scraper

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Provides a Python client for interacting with the Deepweb Scraper API. Includes methods for executing missions, adding documents, and checking system health. Requires the 'requests' library.

```python
import requests

class IntelligenceAgentClient:
    def __init__(self, base_url):
        self.base_url = base_url.rstrip('/')
    
    def execute_mission(self, mission: str) -> dict:
        """Execute intelligence mission"""
        response = requests.post(
            f"{self.base_url}/api/execute_mission",
            json={"mission": mission},
            timeout=300
        )
        response.raise_for_status()
        return response.json()
    
    def add_document(self, text: str, url: str = "", title: str = "") -> dict:
        """Add document to knowledge base"""
        response = requests.post(
            f"{self.base_url}/api/add_document",
            json={"text": text, "url": url, "title": title}
        )
        response.raise_for_status()
        return response.json()
    
    def health_check(self) -> dict:
        """Check system health"""
        response = requests.get(f"{self.base_url}/health")
        response.raise_for_status()
        return response.json()

# Initialize client
client = IntelligenceAgentClient("https://your-url.modal.run")

# Execute mission
result = client.execute_mission(
    "Investigate LockBit ransomware activities"
)

print(result['final_report'])

# Add document
client.add_document(
    text="APT29 is a sophisticated threat actor...",
    url="https://example.com/apt29",
    title="APT29 Analysis"
)

# Health check
health = client.health_check()
print(f"System status: {health['status']}")
print(f"Documents: {health['documents']}")
```

--------------------------------

### Configure Modal Function Settings

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

This snippet shows how to configure the settings for a Modal function in `app.py`. It includes parameters for CPU, memory, timeout, keep-warm instances, and container idle timeout. These settings are crucial for optimizing the performance and resource allocation of the deployed function.

```python
@app.function(
    cpu=2.0,                       # CPU cores (0.1-16)
    memory=4096,                   # RAM in MB (128-65536)
    timeout=900,                   # Max execution time in seconds
    keep_warm=1,                   # Containers to keep warm
    container_idle_timeout=300     # Idle timeout in seconds
)
```

--------------------------------

### Environment Variables Configuration

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Lists essential environment variables for configuring the deepweb-scraper project. It includes required API keys and optional settings for caching, vector dimensions, and sentence transformer models.

```bash
# Required
CEREBRAS_API_KEY=sk-xxxxx          # Primary API key
CEREBRAS_API_KEY_2=sk-yyyyy        # Backup key (optional)

# Optional
CACHE_DIR=/cache                   # Cache directory
VECTOR_DIMENSION=384               # Embedding dimension
MODEL_NAME=all-MiniLM-L6-v2       # Sentence transformer model
```

--------------------------------

### System Metrics via REST API

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Fetches detailed system metrics, including version information. This endpoint provides a timestamp, system status, uptime, document count, cache size, and the system version.

```bash
curl "https://your-app.modal.run/metrics"

# Response:
# {
#   "timestamp": 1699900000.0,
#   "system": {
#     "status": "healthy",
#     "uptime": 3600.5,
#     "documents": 150,
#     "cache_size": 1024000
#   },
#   "version": "2.0.7-final"
# }
```

--------------------------------

### POST /api/add_document

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Adds a new document to the knowledge base. Requires text content and optionally accepts a URL and title.

```APIDOC
## POST /api/add_document

### Description
Add document to knowledge base.

### Method
POST

### Endpoint
/api/add_document

### Parameters
#### Request Body
- **text** (string) - Required - The content of the document.
- **url** (string) - Optional - The URL of the document source.
- **title** (string) - Optional - The title of the document.

### Request Example
```json
{
  "text": "Your document content here...",
  "url": "https://source-url.com",
  "title": "Document Title"
}
```

### Response
#### Success Response (200)
- **status** (string) - The status of the operation (e.g., "success").
- **doc_id** (integer) - The unique identifier assigned to the newly added document.
- **message** (string) - A confirmation message.

#### Response Example
```json
{
  "status": "success",
  "doc_id": 42,
  "message": "Document added with ID 42"
}
```
```

--------------------------------

### Report Generation Prompt

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Constructs a prompt for the 'Analyst' AI to generate a report based on an execution log and a given mission. The prompt specifies the desired report structure, including an executive summary, key findings, detailed analysis, and conclusions.

```python
report_prompt = f"""
Analyze execution log and create report for: '{mission}'

Execution Log:
{json.dumps(execution_log, indent=2)}

Structure:
1. Executive Summary (3-4 sentences)
2. Key Findings (bullet points)
3. Detailed Analysis (2-3 paragraphs)
4. Conclusions

Use professional, analytical tone.
"""

# Model: Qwen Thinking (with chain-of-thought)
report = cerebras.chat.completions.create(
    model="qwen-3-235b-thinking-2507",
    messages=[{"role": "user", "content": report_prompt}],
    temperature=0.5
)
```

--------------------------------

### Add Document to Knowledge Base via REST API

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Adds a new document to the system's vector database for semantic search retrieval. It accepts a JSON payload containing the document's text, an optional URL, and a title.

```bash
curl -X POST "https://your-app.modal.run/api/add_document" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "APT29, also known as Cozy Bear, is a sophisticated threat actor believed to be associated with Russian intelligence services. The group has been active since at least 2008 and is known for conducting long-term espionage operations targeting government, diplomatic, and think tank entities. APT29 employs advanced techniques including zero-day exploits, supply chain compromises, and custom malware.",
    "url": "https://example.com/apt29-analysis",
    "title": "APT29 Threat Profile"
  }'

# Response:
# {
#   "status": "success",
#   "doc_id": 42,
#   "message": "Document added with ID 42"
# }
```

--------------------------------

### Add Document to Knowledge Base (API)

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Adds a document to the knowledge base via a POST request to the /api/add_document endpoint. Requires a JSON payload with 'text', and optionally 'url' and 'title'. Returns the status and ID of the added document.

```bash
curl -X POST https://your-url.modal.run/api/add_document \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Your document content here...",
    "url": "https://source-url.com",
    "title": "Document Title"
  }'
```

--------------------------------

### POST /api/add_document

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Adds a new document to the vector database for semantic search retrieval.

```APIDOC
## POST /api/add_document

### Description
Adds a new document to the vector database for semantic search retrieval.

### Method
POST

### Endpoint
/api/add_document

### Parameters
#### Request Body
- **text** (string) - Required - The content of the document.
- **url** (string) - Optional - The URL of the document.
- **title** (string) - Optional - The title of the document.

### Request Example
```json
{
  "text": "APT29, also known as Cozy Bear, is a sophisticated threat actor believed to be associated with Russian intelligence services. The group has been active since at least 2008 and is known for conducting long-term espionage operations targeting government, diplomatic, and think tank entities. APT29 employs advanced techniques including zero-day exploits, supply chain compromises, and custom malware.",
  "url": "https://example.com/apt29-analysis",
  "title": "APT29 Threat Profile"
}
```

### Response
#### Success Response (200)
- **status** (string) - The status of the operation (e.g., "success").
- **doc_id** (integer) - The unique identifier assigned to the added document.
- **message** (string) - A confirmation message.

#### Response Example
```json
{
  "status": "success",
  "doc_id": 42,
  "message": "Document added with ID 42"
}
```
```

--------------------------------

### JSON Format for Document Import

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Specifies the JSON structure required for bulk document import using the CLI tool. Each object in the array represents a document with 'title', 'url', and 'text' fields.

```json
[
  {
    "title": "Document Title",
    "url": "https://source-url.com",
    "text": "Full document content here..."
  }
]
```

--------------------------------

### Summarize Text with Llama Maverick AI in Python

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

The 'summarize_with_maverick' function uses the Llama Maverick AI model to generate a concise summary of a given text. It accepts a string (up to 8000 characters) and returns a single-paragraph summary. This is ideal for executive summaries and quick information extraction.

```python
def summarize_with_maverick(text: str) -> str:
    # ... implementation details ...
    pass

# Example Usage:
# Summarize a long document
long_doc = agent.get_document_content(5)['content']
summary = agent.summarize_with_maverick(long_doc)
```

--------------------------------

### Execute Mission Endpoint

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

Executes a mission by providing a mission description.

```APIDOC
## POST /api/execute_mission

### Description
Executes a mission by providing a mission description. The system will process the mission and return a report and execution log.

### Method
POST

### Endpoint
/api/execute_mission

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **mission** (string) - Required - The description of the mission to execute.

### Request Example
```json
{
  "mission": "Search for information about ransomware"
}
```

### Response
#### Success Response (200)
- **report** (string) - The generated report for the mission.
- **execution_log** (string) - The detailed execution log of the mission.

#### Response Example
```json
{
  "report": "This is the generated report...",
  "execution_log": "Step 1: Planning\nStep 2: Execution..."
}
```
```

--------------------------------

### Configure Telegram Bot Settings

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

This snippet details the configuration for the Telegram bot in `telegram_bot.py`. It includes the bot's API token, the base URL for its API endpoint, rate limiting settings, and a list of administrator user IDs. These parameters are essential for the bot's operation and security.

```python
TELEGRAM_TOKEN = "YOUR_BOT_TOKEN"
API_BASE_URL = "https://your-url.modal.run"
RATE_LIMIT_SECONDS = 60           # Rate limit per user
ADMIN_USER_IDS = [123456789]      # Admin user IDs
```

--------------------------------

### IntelligenceAgentClient API Endpoints

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

This section details the methods available in the Python client for interacting with the Intelligence Agent API.

```APIDOC
## POST /api/execute_mission

### Description
Execute an intelligence mission and return the report.

### Method
POST

### Endpoint
/api/execute_mission

### Parameters
#### Request Body
- **mission** (string) - Required - The description of the mission to execute.

### Request Example
```json
{
  "mission": "Analyze Log4Shell vulnerability exploitation in the wild"
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the mission was successful.
- **final_report** (string) - The generated report from the mission.
- **metadata** (object) - Contains metadata about the mission execution.
  - **total_duration_seconds** (float) - The total time taken for the mission in seconds.

#### Response Example
```json
{
  "success": true,
  "final_report": "## Executive Summary\nAPT29 has demonstrated increased sophistication...",
  "metadata": {
    "total_duration_seconds": 5.6
  }
}
```

## POST /api/add_document

### Description
Add a document to the knowledge base.

### Method
POST

### Endpoint
/api/add_document

### Parameters
#### Request Body
- **text** (string) - Required - The content of the document.
- **url** (string) - Optional - The URL where the document was sourced from.
- **title** (string) - Optional - The title of the document.

### Request Example
```json
{
  "text": "Log4Shell (CVE-2021-44228) is a critical remote code execution vulnerability...",
  "url": "https://example.com/log4shell",
  "title": "Log4Shell Vulnerability Analysis"
}
```

### Response
#### Success Response (200)
- **id** (integer) - The ID of the added document.
- **message** (string) - A confirmation message.

#### Response Example
```json
{
  "id": 0,
  "message": "Document added successfully."
}
```

## GET /health

### Description
Check system health status.

### Method
GET

### Endpoint
/health

### Parameters
None

### Response
#### Success Response (200)
- **status** (string) - The health status of the system (e.g., 'ok').
- **documents** (integer) - The number of documents currently in the knowledge base.

#### Response Example
```json
{
  "status": "ok",
  "documents": 150
}
```
```

--------------------------------

### Telegram Bot Integration

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Details on configuring and using the Telegram bot for mobile access to the Intelligence Agent.

```APIDOC
## Telegram Bot Commands and Direct Messaging

### Description
Interact with the Intelligence Agent via a Telegram bot, offering mobile access with rate limiting and inline keyboards.

### Configuration

Set the following environment variables or constants in `telegram_bot.py`:

- **`TELEGRAM_TOKEN`**: Your Telegram Bot API token.
- **`API_BASE_URL`**: The base URL of the Intelligence Agent API (e.g., `https://your-app.modal.run`).
- **`RATE_LIMIT_SECONDS`**: The time in seconds users must wait between submitting missions (default: 60).
- **`ADMIN_USER_IDS`**: A list of Telegram user IDs with administrative privileges.

### Starting the Bot

Run the bot script from your terminal:

```bash
python telegram_bot.py
```

### Bot Commands

- **`/start`**: Displays a welcome message and basic help.
- **`/help`**: Provides a detailed usage guide with examples.
- **`/mission <description>`**: Executes an intelligence mission with the provided description.
- **`/status`**: Checks the health status of the Intelligence Agent system.
- **`/stats`**: Shows the user's personal usage statistics.

### Direct Messaging

Users can send direct messages (without commands) to initiate intelligence missions. The bot will process the message as a mission request.

**Example Interaction:**

*User:* `Investigate APT29 Cozy Bear activities in 2024`

*Bot:* `Executing Mission...`
*Bot:* `Mission Complete`
     `## Executive Summary`
     `APT29 has demonstrated increased sophistication...`
     `## Key Findings`
     `- Enhanced use of legitimate cloud services for C2`
     `- Shift to stealthier, longer-term operations`
     `...`
     `Completed in 5.6s | 3/3 steps successful`

### Rate Limiting

If a user attempts to submit another mission before the `RATE_LIMIT_SECONDS` period has elapsed, the bot will respond with a rate limiting message.

**Example Rate Limiting Response:**

*Bot:* `Please wait 45s before submitting another mission.`
```

--------------------------------

### Summarize with Maverick API

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

Summarizes provided text using the Cerebras AI Llama Maverick model. It supports long texts and includes automatic retry and fallback mechanisms.

```APIDOC
## POST /summarize_with_maverick

### Description
Summarizes long text using Cerebras AI's Llama Maverick model with automatic retry and fallback.

### Method
POST

### Endpoint
/summarize_with_maverick

#### Query Parameters
This endpoint does not have query parameters.

#### Request Body
- **text** (string) - Required - The text to summarize (maximum 8000 characters).

### Request Example
```python
system = IntelligenceSystem()

long_document = """
APT29, also known as Cozy Bear, is a sophisticated threat actor believed to be
associated with Russian intelligence services. The group has been active since
at least 2008 and is known for conducting long-term espionage operations targeting
government, diplomatic, and think tank entities. APT29 employs advanced techniques
including zero-day exploits, supply chain compromises, and custom malware. Recent
campaigns show APT29 leveraging legitimate cloud services for command and control,
making detection more difficult. The group has demonstrated patience in operations,
often maintaining access for months before exfiltrating data.
"""

summary = system.summarize_with_maverick(long_document)
```

### Response
#### Success Response (200)
- **summary** (string) - A concise one-paragraph summary of the input text.

#### Response Example
```json
{
  "summary": "APT29 (Cozy Bear) is a Russian-linked threat actor active since 2008, specializing in long-term espionage against government and diplomatic targets using zero-days, supply chain attacks, and cloud-based C2 infrastructure."
}
```
```

--------------------------------

### Python Intelligence Agent Client

Source: https://context7.com/oktetod/deepweb-scraper/llms.txt

A Python client for interacting with the Intelligence Agent REST API. It allows executing missions, adding documents, and checking system health. Requires the 'requests' library.

```python
import requests
from typing import Dict, Any, Optional

class IntelligenceAgentClient:
    """Python SDK client for Intelligence Agent System API."""

    def __init__(self, base_url: str, timeout: int = 300):
        self.base_url = base_url.rstrip('/')
        self.timeout = timeout

    def execute_mission(self, mission: str) -> Dict[str, Any]:
        """Execute an intelligence mission and return the report."""
        response = requests.post(
            f"{self.base_url}/api/execute_mission",
            json={"mission": mission},
            timeout=self.timeout
        )
        response.raise_for_status()
        return response.json()

    def add_document(self, text: str, url: str = "", title: str = "") -> Dict[str, Any]:
        """Add a document to the knowledge base."""
        response = requests.post(
            f"{self.base_url}/api/add_document",
            json={"text": text, "url": url, "title": title},
            timeout=30
        )
        response.raise_for_status()
        return response.json()

    def health_check(self) -> Dict[str, Any]:
        """Check system health status."""
        response = requests.get(f"{self.base_url}/health", timeout=10)
        response.raise_for_status()
        return response.json()

# Usage example
client = IntelligenceAgentClient("https://your-app.modal.run")

# Check system health
health = client.health_check()
print(f"Status: {health['status']}, Documents: {health['documents']}")

# Add documents to knowledge base
client.add_document(
    text="Log4Shell (CVE-2021-44228) is a critical remote code execution vulnerability...",
    url="https://example.com/log4shell",
    title="Log4Shell Vulnerability Analysis"
)

# Execute an investigation
result = client.execute_mission("Analyze Log4Shell vulnerability exploitation in the wild")
if result["success"]:
    print(result["final_report"])
    print(f"Completed in {result['metadata']['total_duration_seconds']:.1f}s")

```

--------------------------------

### Vulnerability Assessment Report Snippet

Source: https://github.com/oktetod/deepweb-scraper/blob/main/readme.md

A snippet of a Markdown report detailing critical vulnerabilities in Apache products, including specific CVEs, severity levels, and potential impacts.

```markdown
## Latest Apache Critical Vulnerabilities

### CVE-2023-XXXXX (Apache HTTP Server)
Severity: Critical (CVSS 9.8)
Impact: Remote Code Execution...

### CVE-2023-YYYYY (Apache Log4j)
Severity: High (CVSS 8.1)
Impact: Information Disclosure...
```