### Copy Example Config and Restart Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Copy the example configuration file and restart the stack to apply default settings. ```bash cp config.example.yaml config.yaml # 修正 secret_key docker compose restart ``` -------------------------------- ### Build and Start Stack with Build Option Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md A single command to build the adapter image and start the stack. ```bash docker compose up -d --build ``` -------------------------------- ### Copy Example Config and Restart Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/operations.md If SearXNG crashes on startup due to a missing `config.yaml`, copy the example configuration file and edit the `secret_key`. Then restart the Docker Compose services. ```bash cp config.example.yaml config.yaml # edit secret_key docker compose restart ``` -------------------------------- ### Prepare Configuration File Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Copy the example configuration file and update the secret key. ```bash cp config.example.yaml config.yaml ``` ```yaml server: secret_key: "YOUR_RANDOM_KEY_AT_LEAST_32_CHARACTERS" ``` ```bash # any of the three python3 -c "import secrets; print(secrets.token_hex(32))" openssl rand -hex 32 head -c 32 /dev/urandom | xxd -p -c 32 ``` -------------------------------- ### Start Docker Stack Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Start the Searcharvester services using Docker Compose. The first run may take longer due to image pulls. ```bash docker compose up -d ``` ```bash docker compose ps ``` -------------------------------- ### Update SearXNG Configuration Example Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Example of updating SearXNG settings, noting potential changes in release notes. ```yaml settings.yml ``` -------------------------------- ### POST /search Request Example Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md This example demonstrates how to make a POST request to the /search endpoint to find the latest AI news using specific search engines and categories. Ensure the Content-Type header is set to application/json. ```bash curl -X POST http://localhost:8000/search \ -H 'Content-Type: application/json' \ -d '{ "query": "latest AI news", "max_results": 5, "engines": "duckduckgo,brave", "categories": "news" }' ``` -------------------------------- ### Start Searcharvester Stack in Background Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Use this command to start all services in detached mode. ```bash docker compose up -d ``` -------------------------------- ### Production Checklist: Configure Reverse Proxy Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Example of a Caddyfile for setting up a reverse proxy with TLS. ```conf Caddyfile ``` -------------------------------- ### Production Checklist: Log Rotation Settings Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Example of log rotation settings, noting they might be too small for production. ```yaml log: max-size: 1m max-file: 1 ``` -------------------------------- ### Set up Adapter Development Environment Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Prepare the development environment for the adapter by setting up a virtual environment, installing dependencies, and configuring the adapter to use a published SearXNG instance. ```bash # Keep SearXNG in Docker (or leave it as-is) cd simple_tavily_adapter python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt ``` ```yaml adapter: searxng_url: "http://localhost:8999" # instead of http://searxng:8080 ``` ```bash uvicorn main:app --reload --port 8000 ``` -------------------------------- ### Clone and Configure Searcharvester Source: https://github.com/vakovalskii/searcharvester/blob/main/README.md Clone the repository, navigate to the directory, and copy the example configuration file. Ensure you change the server.secret_key in the configuration. ```bash git clone git@github.com:vakovalskii/searcharvester.git cd searcharvester cp config.example.yaml config.yaml # Change server.secret_key (32+ chars) ``` -------------------------------- ### Build and Run Tavily Adapter Source: https://github.com/vakovalskii/searcharvester/blob/main/README.md Builds and starts the Tavily adapter service in detached mode. Use this for fast iteration during development. ```bash cd simple_tavily_adapter docker compose build tavily-adapter && docker compose up -d ``` -------------------------------- ### Test Search Endpoint Source: https://github.com/vakovalskii/searcharvester/blob/main/README.md Send a POST request to the /search endpoint to test its functionality. This example queries for 'bitcoin price' and requests a maximum of 3 results. ```bash curl -X POST localhost:8000/search -H 'Content-Type: application/json' \ -d '{"query":"bitcoin price","max_results":3}' ``` -------------------------------- ### Run Smoke Test on Host Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Execute the smoke test client directly on the host machine, assuming dependencies are installed. ```bash cd simple_tavily_adapter && python test_client.py ``` -------------------------------- ### Perform Fast Search and Get Snippets Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md Use this pattern for quick searches where you need the titles, URLs, and content snippets of the top results. ```bash curl -sX POST localhost:8000/search -H 'Content-Type: application/json' \ -d '{"query":"what is RAG","max_results":5}' \ | jq '.results[] | {title, url, content}' ``` -------------------------------- ### Search Result JSON Output Source: https://github.com/vakovalskii/searcharvester/blob/main/hermes_skills/searcharvester-search/SKILL.md Example of the JSON structure returned by the search endpoint, including the query, and a list of results with URL, title, and content snippet. ```json { "query": "...", "results": [ { "url": "https://...", "title": "...", "content": "short snippet from the search engine" } ] } ``` -------------------------------- ### Tavily-Compatible Search Endpoint Source: https://github.com/vakovalskii/searcharvester/blob/main/README.md Use this Python snippet to interact with the `/search` endpoint, which is a drop-in replacement for the Tavily API. Ensure you have the Tavily client installed and provide your API key and base URL. ```python from tavily import TavilyClient client = TavilyClient(api_key="ignored", base_url="http://localhost:8000") response = client.search(query="...", max_results=5, include_raw_content=True) ``` -------------------------------- ### Researcher Template: Search and Extract Workflow Source: https://github.com/vakovalskii/searcharvester/blob/main/hermes_skills/searcharvester-deep-research/SKILL.md This template defines the role and methodology for a Researcher agent. It outlines the process of investigating a sub-question, using search and extract scripts, and formatting findings. The agent must start with a search query and is constrained to using only terminal tools. ```bash python3 /opt/data/skills/searcharvester-search/scripts/search.py \ --query "" --max-results 5 ``` ```bash python3 /opt/data/skills/searcharvester-extract/scripts/extract.py \ --url "" ``` ```bash grep -ni 'keyword' ./extracts/.md ``` ```bash head -200 ./extracts/.md ``` ```bash sed -n '300,600p' ./extracts/.md ``` -------------------------------- ### GET /health Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md Performs a health check of the API. ```APIDOC ## GET /health ### Description Performs a health check of the API. ### Method GET ### Endpoint /health ### Response (Response details not explicitly provided in the source, but typically a status indicator.) ### Error codes (Error codes not explicitly provided in the source.) ``` -------------------------------- ### Build Adapter Image and Restart Stack Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Manually rebuild the adapter image and restart the stack after code changes. ```bash docker compose build tavily-adapter docker compose up -d ``` -------------------------------- ### GET /extract/{id}/{page} Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md Retrieves the next page of extracted content, specifically for full-size extractions. ```APIDOC ## GET /extract/{id}/{page} ### Description Retrieves the next page of extracted content, specifically for full-size extractions. ### Method GET ### Endpoint /extract/{id}/{page} ### Parameters #### Path Parameters - **id** (string) - Required - The ID of the extraction job. - **page** (int) - Required - The page number to retrieve. ### Response (Response details not explicitly provided in the source, but would typically include the markdown content for the requested page.) ### Error codes (Error codes not explicitly provided in the source.) ``` -------------------------------- ### GET /extract/{id}/{page} Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md Retrieves a specific page of extracted content using the extraction ID and page number. ```APIDOC ## GET /extract/{id}/{page} ### Description Retrieves a specific page of extracted content using the extraction ID and page number. ### Method GET ### Endpoint /extract/{id}/{page} ### Parameters #### Path Parameters - **id** (string) - Required - The unique identifier for the extraction. - **page** (integer) - Required - The page number to retrieve. ### Response #### Success Response (200) - **content** (string) - The extracted content for the specified page. #### Response Example ```json { "content": "Extracted content from page X..." } ``` ``` -------------------------------- ### System Context Diagram (C4 Model) Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/architecture.md Illustrates the SearXNG Tavily Adapter's external interactions with developers and search engines. Rendered using Mermaid. ```mermaid C4Context title System Context — SearXNG Tavily Adapter Person(dev, "Developer / LLM agent", "Sends search requests in Tavily API format") System(stack, "SearXNG Tavily Adapter", "Self-hosted stack: SearXNG + FastAPI adapter,\
Tavily-compatible API") System_Ext(engines, "Search engines", "Google, DuckDuckGo, Brave, Bing,\
Startpage, etc. (queried by SearXNG)") System_Ext(sites, "Target sites", "HTML pages from search results\
(scraped when include_raw_content=true)") Rel(dev, stack, "POST /search", "HTTP / JSON") Rel(stack, engines, "Search queries", "HTTPS") Rel(stack, sites, "GET pages (scraping)", "HTTPS") UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="1") ``` -------------------------------- ### GET /health Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md Checks the health status of the Search Arvester service. This endpoint is used by Docker Compose for health checks. ```APIDOC ## GET /health ### Description Checks the health status of the Search Arvester service. This endpoint is used by Docker Compose for health checks. ### Method GET ### Endpoint `/health` ### Response #### Success Response (200) ```json {"status":"ok","service":"searxng-tavily-adapter","version":"2.0.0"} ``` ``` -------------------------------- ### View All Service Logs (Follow Mode) Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Stream logs from all services in real-time. ```bash docker compose logs -f ``` -------------------------------- ### Pull Image Updates and Rebuild/Restart Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Pull the latest images for specified services, rebuild the adapter if necessary, and restart the stack. ```bash docker compose pull searxng redis # 拉取更新 docker compose build tavily-adapter # 改过适配器时 docker compose up -d ``` -------------------------------- ### Tavily Adapter Search Response Shape Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Example of the expected JSON response structure for a search query made to the Tavily Adapter. ```json { "query": "bitcoin price", "results": [ { "url": "...", "title": "...", "content": "...", "score": 0.9, "raw_content": null } ], "response_time": 1.23, "request_id": "..." } ``` -------------------------------- ### Create Research Plan File Source: https://github.com/vakovalskii/searcharvester/blob/main/hermes_skills/searcharvester-deep-research/SKILL.md Generates a plan.md file with intent, sub-questions, critical facts, and out-of-scope sections. Use this to structure the research approach. ```bash cat > ./plan.md << 'EOF' ## Intent ## Sub-questions (2–3) 1. 2. ... ## Critical facts to fact-check - ## Out of scope - EOF ``` -------------------------------- ### Run Built-in Adapter Smoke Test Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Execute the smoke test client located within the adapter container. ```bash docker compose exec tavily-adapter python test_client.py ``` -------------------------------- ### Production Checklist: Enable Limiter Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Enable rate limiting in the adapter configuration for production environments. ```yaml limiter: true ``` -------------------------------- ### Integrate with Raw HTTP Requests Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Make direct HTTP POST requests to the adapter's search endpoint using a library like `requests`. ```python import requests r = requests.post("http://localhost:8000/search", json={ "query": "...", "max_results": 5, "include_raw_content": True, }) r.raise_for_status() data = r.json() ``` -------------------------------- ### Run Tests for Tavily Adapter Source: https://github.com/vakovalskii/searcharvester/blob/main/README.md Executes the tests for the Tavily adapter within the Docker environment. The -q flag provides quiet output. ```bash docker compose exec tavily-adapter pytest -q ``` -------------------------------- ### Test Extract Endpoint Source: https://github.com/vakovalskii/searcharvester/blob/main/README.md Send a POST request to the /extract endpoint to convert a URL into clean markdown. This example extracts content from a Wikipedia page with a medium size preset. ```bash curl -X POST localhost:8000/extract -H 'Content-Type: application/json' \ -d '{"url":"https://en.wikipedia.org/wiki/Docker_(software)","size":"m"}' ``` -------------------------------- ### GET /extract/{id}/{page} Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md Retrieves a specific page of previously extracted content for a given document ID. This endpoint is intended for use after an initial extraction request with `size=f`. ```APIDOC ## GET /extract/{id}/{page} ### Description Returns a page of previously extracted content. Only works for documents originally requested with `size=f`. ### Method GET ### Endpoint `/extract/{id}/{page}` ### Parameters #### Path Parameters - **id** (string (16 hex)) - Required - `id` from the previous `POST /extract` - **page** (int ≥ 1) - Required - page number ### Response #### Success Response (200) Response has the same shape as `POST /extract` with `size=f`, but with `pages.current = page`. #### Error Codes - **200** - ok - **404** - `id` not found (or `page` > `total`) - **422** - invalid path parameters ``` -------------------------------- ### Clone the Repository Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Clone the Searcharvester repository and navigate into the project directory. ```bash git clone git@github.com:vakovalskii/searcharvester.git cd searcharvester ``` -------------------------------- ### Rebuild Adapter and Restart Stack Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Run this command after modifying adapter code to rebuild the adapter image and restart the stack. ```bash docker compose build tavily-adapter && docker compose up -d ``` -------------------------------- ### Pull Pre-built Image and Restart Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/operations.md If using a pre-built image from GHCR, pull the latest version and restart the services to apply updates. Use `docker compose pull` followed by `docker compose up -d`. ```bash docker compose pull tavily-adapter docker compose up -d ``` -------------------------------- ### View SearXNG Request Logs Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/operations.md Adapter logs provide request details, response times, and result counts. Useful for monitoring search activity. ```log INFO:main:Search request: bitcoin price INFO:main:Search completed: 3 results in 1.42s ``` -------------------------------- ### Perform a Basic Search Source: https://github.com/vakovalskii/searcharvester/blob/main/hermes_skills/searcharvester-search/SKILL.md Execute a basic web search using the provided query and specify the maximum number of results. ```bash python3 SKILL_DIR/scripts/search.py --query "what is retrieval augmented generation" --max-results 5 ``` -------------------------------- ### Integrate with Local Tavily Client Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Use the local `simple_tavily_adapter` client when your code runs on the same host to avoid HTTP overhead. The client reads configuration from `config.yaml`. ```python from simple_tavily_adapter.tavily_client import TavilyClient client = TavilyClient() # reads config.yaml response = client.search(query="...", max_results=5, include_raw_content=True) ``` -------------------------------- ### Enter a Service Container Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Execute commands inside a specific service's container. ```bash docker compose exec tavily-adapter sh ``` -------------------------------- ### Integrate with Tavily Python Client Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Use the official `tavily-python` client to interact with the adapter. The API key is ignored, and the base URL should point to your running adapter. ```python from tavily import TavilyClient client = TavilyClient( api_key="anything", # ignored by the adapter base_url="http://localhost:8000" # ← your adapter ) response = client.search(query="what is machine learning", max_results=5, include_raw_content=True) ``` -------------------------------- ### Production Checklist: Set SearXNG Base URL Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Configure the base URL for SearXNG in the docker-compose.yaml for production deployment. ```yaml SEARXNG_BASE_URL=your_domain.com ``` -------------------------------- ### Restart Multiple Services Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Restart multiple services simultaneously, useful after configuration changes. ```bash docker compose restart searxng tavily-adapter ``` -------------------------------- ### Configure SearXNG URL in Adapter Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Set the correct SearXNG URL within the adapter's configuration, especially when running the adapter outside Docker. ```yaml adapter: searxng_url: "http://searxng:8080" ``` -------------------------------- ### Extract Full Document with Pagination Source: https://github.com/vakovalskii/searcharvester/blob/main/hermes_skills/searcharvester-extract/SKILL.md For very long documents, use the 'f' preset to paginate content. The first command fetches page 1 and provides an ID for subsequent pages. The second command fetches a specific page using the ID. ```bash # Page 1 (also returns metadata: total pages, id for follow-up pages) python3 SKILL_DIR/scripts/extract.py --url "https://long.example.com/article" --size f ``` ```bash # Fetch page 2 using id from first response python3 SKILL_DIR/scripts/extract.py --id b275618ca10e6c62 --page 2 ``` -------------------------------- ### POST /search Sequence Diagram Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/architecture.md Illustrates the sequence of operations when a client makes a POST /search request. Includes validation, SearXNG interaction, and optional raw content extraction with trafilatura. ```mermaid sequenceDiagram autonumber participant C as Client participant A as FastAPI (main.py) participant S as SearXNG participant W as Target sites C->>A: POST /search {query, max_results, engines?, categories?, include_raw_content?} A->>A: Validation (Pydantic SearchRequest) A->>S: POST /search?format=json& engines=...&categories=... S-->>A: JSON with results[] opt include_raw_content == true par Parallel fetch + trafilatura (asyncio.gather) A->>W: GET url_1 A->>W: GET url_2 A->>W: GET url_N end W-->>A: HTML A->>A: trafilatura.extract( output_format='markdown') end A->>A: Map SearXNG results → TavilyResult[] A-->>C: TavilyResponse (JSON, raw_content as md) ``` -------------------------------- ### Debug SearXNG Connectivity Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Test SearXNG's ability to reach search engines by fetching search results. ```bash docker compose exec searxng wget -qO- "http://localhost:8080/search?q=test&format=json" | head -c 500 ``` -------------------------------- ### SearchArvester Architecture Diagram Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/overview.md Illustrates the interaction between your code, the Tavily Adapter, SearXNG, and Redis/Valkey within the SearchArvester stack. ```text ┌──────────────┐ HTTP/JSON ┌────────────────┐ HTTP/JSON ┌──────────┐ │ Your code / │───────────────▶│ Tavily Adapter │───────────────▶│ SearXNG │ │ LLM / curl │◀───────────────│ (port 8000) │◀───────────────│ (port │ └──────────────┘ Tavily └────────┬───────┘ SearXNG │ 8999) │ format │ └────┬─────┘ │ aiohttp + trafilatura │ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ Target sites │ │ Redis/Valkey │ │ (scraped for │ │ (cache) │ │ markdown) │ └──────────────┘ └──────────────┘ ``` -------------------------------- ### Verify SearXNG API Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Test the SearXNG instance by performing a search query via curl and checking the number of results. ```bash curl "http://localhost:8999/search?q=test&format=json" | jq '.results | length' ``` -------------------------------- ### Tavily Adapter Component Diagram Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/architecture.md Visualizes the components within the Tavily Adapter and their relationships. Shows the FastAPI app, clients, configuration, extraction library, cache, and data models. ```mermaid C4Component title Component — Tavily Adapter (Python) Person(dev, "Client") ContainerDb(searxng_box, "SearXNG", "HTTP service") Container_Ext(sites, "Target sites", "HTTP") Container_Boundary(adapter, "simple_tavily_adapter") { Component(fastapi, "main.py — FastAPI app", "POST /search, POST /extract,\nGET /extract/{id}/{page}, GET /health", "Endpoints and request validation\n(Pydantic: SearchRequest, ExtractRequest)") Component(client, "tavily_client.py", "Python class", "In-process client\n(for scripts without HTTP)") Component(config, "config_loader.py", "Singleton", "Reads config.yaml,\nexposes params via properties") Component(extractor, "trafilatura.extract()", "Python library", "Main-content extraction →\nmarkdown with headings, links,\ntables. Strips nav/footer/ads") Component(cache, "_extract_cache", "in-memory dict + TTL", "id → {url, title, content}.\nTTL 30 min. Backs /extract/{id}/{page}") Component(models, "TavilyResult / TavilyResponse", "Pydantic models", "Response schema for /search (Tavily format)") } Rel(dev, fastapi, "HTTP/JSON requests") Rel(fastapi, config, "reads searxng_url,\nscraper_timeout, user_agent") Rel(fastapi, searxng_box, "POST /search?format=json", "aiohttp") Rel(fastapi, sites, "GET HTML (aiohttp)", "HTTPS") Rel(fastapi, extractor, "HTML → markdown") Rel(fastapi, cache, "put/get by id") Rel(fastapi, models, "builds /search response") Rel(client, config, "uses") Rel(client, extractor, "uses") Rel(client, models, "uses") UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1") ``` -------------------------------- ### Search with Tavily Adapter Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/getting-started.md Perform a search query using the Tavily Adapter API. Specify the query and the maximum number of results. ```bash curl -X POST http://localhost:8000/search \ -H "Content-Type: application/json" \ -d '{"query": "bitcoin price", "max_results": 3}' | jq ``` -------------------------------- ### POST /extract + Pagination Sequence Diagram Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/architecture.md Details the sequence for POST /extract requests, including URL hashing for cache keys, cache lookups, content extraction using trafilatura, and handling paginated content retrieval. ```mermaid sequenceDiagram autonumber participant C as Client participant A as FastAPI participant K as _extract_cache participant W as Target site C->>A: POST /extract {url, size="f"} A->>A: id = md5(url)[:16] A->>K: lookup id alt cache miss A->>W: GET url W-->>A: HTML A->>A: trafilatura.extract(md) A->>K: store id → {url, title, content} (TTL 30 min) else cache hit (< 30 min) K-->>A: content end A->>A: slice content by size (s=5k, m=10k, l=25k, f=25k/page) A-->>C: {id, content, pages: {current, total, next?}} Note over C,A: Content spans multiple pages → client fetches subsequent ones C->>A: GET /extract/{id}/2 A->>K: lookup id K-->>A: content A->>A: slice [25000 : 50000] A-->>C: {content, pages: {current: 2, total: N, next?}} ``` -------------------------------- ### Tavily Adapter File Responsibilities Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/architecture.md Lists the core Python files within the Tavily Adapter and their primary roles, including the FastAPI app, client, configuration loader, and Dockerfile. ```markdown | File | Role | |---|---| | `simple_tavily_adapter/main.py` | FastAPI app. Endpoints: `POST /search`, `POST /extract`, `GET /extract/{id}/{page}`, `GET /health`. Houses trafilatura extractor and in-memory cache | | `simple_tavily_adapter/tavily_client.py` | Python class `TavilyClient`, mirroring `tavily-python` API. For scripts that don't want HTTP | | `simple_tavily_adapter/config_loader.py` | Reads unified `config.yaml`, exposes params via `@property` | | `simple_tavily_adapter/Dockerfile` | `python:3.11-slim` + `curl` for health-check. Runs `uvicorn main:app` | | `simple_tavily_adapter/requirements.txt` | FastAPI, aiohttp, **trafilatura**, **lxml**, pydantic, pyyaml | | `simple_tavily_adapter/test_client.py` | Smoke test for `TavilyClient` | ``` -------------------------------- ### Configure Adapter Scraper Timeout and User Agent Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Adjust the scraper timeout and User-Agent in the adapter configuration to handle website blocking or timeouts. ```yaml adapter: scraper: timeout: 20 user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ..." ``` -------------------------------- ### Container Diagram (C4 Model) Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/architecture.md Details the deployment units (containers) within the SearXNG Tavily Adapter stack, including their technologies and ports. Rendered using Mermaid. ```mermaid C4Container title Container — docker-compose stack Person(dev, "Developer / LLM agent") System_Ext(engines, "Search engines", "Google, DuckDuckGo, ...") System_Ext(sites, "Target sites", "HTML pages") System_Boundary(stack, "SearXNG Tavily Adapter stack") { Container(adapter, "Tavily Adapter", "Python 3.11, FastAPI, aiohttp", "Accepts Tavily-compatible requests,\
proxies to SearXNG,\
optionally scrapes pages.\
Port 8000 (published)") Container(searxng, "SearXNG", "Python, Flask", "Metasearch engine.\
Port 8080 internal → 8999 on host") ContainerDb(redis, "Valkey (Redis)", "valkey:8-alpine", "Cache and state for SearXNG.\
Internal Docker network only") } Rel(dev, adapter, "POST /search", "HTTP/JSON, 8000") Rel(adapter, searxng, "POST /search?format=json", "HTTP, internal network") Rel(adapter, sites, "GET pages\
(when include_raw_content)", "HTTPS") Rel(searxng, engines, "HTTP queries", "HTTPS") Rel(searxng, redis, "Cache / sessions", "RESP (Redis protocol)") UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1") ``` -------------------------------- ### View Single Service Logs Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Stream logs from a specific service in real-time. ```bash docker compose logs -f tavily-adapter ``` -------------------------------- ### Remove Containers and Volumes (Clear Cache) Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/zh/operations.md Use this command to stop and remove services, containers, and volumes, effectively clearing the cache. ```bash docker compose down -v ``` -------------------------------- ### Set up LLM Credentials for Research Endpoint Source: https://github.com/vakovalskii/searcharvester/blob/main/README.md Optionally, configure LLM credentials for the /research endpoint by creating a .env file with your OpenAI API key and base URL. This is required for deep research capabilities. ```bash cat > .env < /workspace/report.md" │ │ │ │ + print "REPORT_SAVED:" │ │ │ │──exit 0 (--rm) │ │ │◀─container done───────│ │ │ │ │──read logs + report.md │ │ │ │─ check REPORT_SAVED marker │ │ │ │─ status = completed │ │ │ │ │ (polling in parallel) │ │──GET /research/{id}────▶│ │ │◀─200 {completed, report}│ │ ``` -------------------------------- ### Tavily Adapter Configuration Parameters Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/architecture.md Specifies configuration parameters read by the Tavily Adapter code, such as SearXNG URL, server host/port, and scraper timeout. Notes parameters that are hardcoded and not read from config. ```markdown - `adapter.searxng_url` → where the adapter calls - `adapter.server.host`, `adapter.server.port` → uvicorn bind - `adapter.scraper.timeout` → per-page timeout - `adapter.scraper.max_content_length` → `raw_content` length limit - `adapter.scraper.user_agent` → User-Agent for scraping **Not read by the code (hardcoded)**: `adapter.search.default_engines`, `default_categories`, `default_language`, `safesearch`, `default_max_results`. They exist as properties in `config_loader.py` but aren't applied in `main.py`. Known wart — see [`../../CLAUDE.md`](../../CLAUDE.md). ``` -------------------------------- ### Critic Template: Adversarial Search and Evaluation Source: https://github.com/vakovalskii/searcharvester/blob/main/hermes_skills/searcharvester-deep-research/SKILL.md This template defines the role and methodology for a Critic agent, designed to challenge findings from a Researcher. It details how to perform adversarial searches based on existing claims, extract relevant information, and present counter-evidence or a final verdict. ```bash cat ./extracts/.md | grep ... ``` -------------------------------- ### Deep Read Article with Pagination Source: https://github.com/vakovalskii/searcharvester/blob/main/docs/en/api.md For long articles, this pattern extracts the first page and then iteratively fetches the remaining pages using the provided ID and total page count. ```bash # First page RESP=$(curl -sX POST localhost:8000/extract -H 'Content-Type: application/json' \ -d '{"url":"https://en.wikipedia.org/wiki/Linux","size":"f"}') echo "$RESP" | jq -r '.content' ID=$(echo "$RESP" | jq -r '.id') TOTAL=$(echo "$RESP" | jq -r '.pages.total') # Remaining pages for P in $(seq 2 $TOTAL); do curl -s localhost:8000/extract/$ID/$P | jq -r '.content' done ```