### llms.txt Quick Start Example

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/index.mdx

This example demonstrates the basic structure of an llms.txt file. It includes a project title, a brief description, and sections for essential documentation links and optional resources. This format helps LLMs quickly understand the site's structure and key content.

```markdown
# Project Name

> Brief project description in one or two lines

## Documentation

- [Quick Start](https://example.com/quickstart): Start here
- [API Reference](https://example.com/api): Complete API description
- [Examples](https://example.com/examples): Ready-made solutions

## Optional

- [Changelog](https://example.com/changelog): Version history
- [Contributing](https://example.com/contributing): How to contribute
```

--------------------------------

### Create and Run a Simple Web App with @framework/core

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/examples/advanced.md

A basic example demonstrating how to create a web application using the @framework/core library. It sets up a simple route for the homepage and starts a server on port 3000.

```javascript
import { createApp } from '@framework/core'

const app = createApp()

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(3000)
```

--------------------------------

### Manual llms.txt Structure (Markdown)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/guides/getting-started.md

This markdown snippet demonstrates the basic structure of an llms.txt file, including a project description and categorized links to important pages. It serves as a template for manually creating the file.

```markdown
# My Project

> Brief project description — what it does and who it's for.

Additional information about the project to help LLMs understand the context.

## Documentation

- [Quick Start](https://example.com/docs/quickstart): Step-by-step guide for new users
- [Installation](https://example.com/docs/installation): Requirements and installation instructions
- [API Reference](https://example.com/docs/api): Complete description of all endpoints

## Guides

- [Authentication](https://example.com/guides/auth): How to set up authorization
- [Deploy](https://example.com/guides/deploy): Production deployment

## Optional

- [Changelog](https://example.com/changelog): Version history
- [Contributing](https://example.com/contributing): How to contribute to the project

```

--------------------------------

### Integrate starlight-llms-txt with Astro (JavaScript)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/guides/getting-started.md

This JavaScript snippet shows how to install and configure the `starlight-llms-txt` plugin within an Astro project's configuration file (`astro.config.mjs`). It enables automatic generation and inclusion of llms.txt for Starlight sites.

```javascript
// astro.config.mjs
import starlightLlmsTxt from 'starlight-llms-txt'

export default defineConfig({
  integrations: [
    starlight({
      plugins: [starlightLlmsTxt()],
    }),
  ],
})

```

--------------------------------

### Install and Run llmstxt CLI

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Demonstrates how to install the llmstxt CLI globally using npm or run it directly using npx. This is the primary method for interacting with the tool.

```bash
# Global installation
npm install -g llmstxt

# Or run via npx (no installation)
npx llmstxt gen https://example.com/sitemap.xml
```

--------------------------------

### Generate llms.txt using llmstxt CLI (Bash)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/guides/getting-started.md

These bash commands demonstrate how to use the llmstxt command-line tool to generate an llms.txt file. The first command generates a basic llms.txt from a sitemap, while the second generates a more detailed version with full content.

```bash
# Generate from sitemap.xml
npx llmstxt gen https://example.com/sitemap.xml > llms.txt

# Generate with full content
npx llmstxt gen-full https://example.com/sitemap.xml > llms-full.txt

```

--------------------------------

### Install llmstxt-architect with uvx and pip

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Provides instructions for quick installation using uvx and standard installation via pip. uvx is a tool for managing and running applications.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uvx --from llmstxt-architect llmstxt-architect --help

pip install llmstxt-architect
```

--------------------------------

### Install @framework/core using npm

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/examples/advanced.md

This snippet shows the command to install the core framework package using npm. It's a prerequisite for using the framework.

```bash
npm install @framework/core
```

--------------------------------

### Self-Host llm.codes Project

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llm-codes.md

This code snippet outlines the steps to self-host the llm.codes project. It involves cloning the repository, installing dependencies using npm, and starting the development server. Environment variables are also mentioned as necessary for configuration.

```bash
git clone https://github.com/amantus-ai/llm-codes
cd llm-codes
npm install
npm run dev
```

--------------------------------

### Full llmstxt Command Example

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

A comprehensive example combining multiple options like path filtering, title replacement, custom metadata, and concurrency control to generate 'llms.txt'.

```bash
npx llmstxt@latest gen https://docs.example.com/sitemap.xml \
  -ep "**/blog/**" \
  -ep "**/changelog/**" \
  -ep "**/privacy" \
  -ep "**/terms" \
  -rt 's/| Docs//' \
  -t 'Example Docs' \
  -d 'Official documentation for Example' \
  -c 5 \
  > llms.txt
```

--------------------------------

### Basic Usage with OpenAI

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example demonstrates how to use llmstxt-architect with OpenAI models for description generation.

```APIDOC
## Basic Usage with OpenAI

### Description
Uses OpenAI models to generate descriptions for provided URLs.

### Method
CLI Command

### Endpoint
N/A (CLI Tool)

### Parameters
#### Query Parameters
- **--urls** (string) - Required - URL to process
- **--max-depth** (integer) - Optional - Crawl depth (1-5), Default: 5
- **--llm-name** (string) - Optional - Model name
- **--llm-provider** (string) - Optional - Provider (anthropic, openai, ollama), Default: anthropic
- **--project-dir** (string) - Optional - Results directory, Default: llms_txt
- **--output-file** (string) - Optional - Output file name, Default: llms.txt
- **--blacklist-file** (string) - Optional - File with URLs to exclude
- **--extractor** (string) - Optional - Extraction method (default, bs4), Default: default

### Request Example
```bash
export OPENAI_API_KEY=sk-...

llmstxt-architect \
  --urls https://docs.example.com \
  --max-depth 2 \
  --llm-name gpt-4o \
  --llm-provider openai \
  --project-dir output
```

### Response
#### Success Response
Generates an `llms.txt` file in the specified `project-dir`.

#### Response Example
```text
# llms.txt content will be generated here
```
```

--------------------------------

### Update from Local File

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example shows how to update descriptions using a local `llms.txt` file as a source.

```APIDOC
## Update from Local File

### Description
Updates descriptions in an existing `llms.txt` file, using a local file as the source for existing content and structure.

### Method
CLI Command

### Endpoint
N/A (CLI Tool)

### Parameters
#### Query Parameters
- **--existing-llms-file** (string) - Required - Path to the local `llms.txt` file.
- **--update-descriptions-only** (boolean) - Required - Flag to indicate description update mode.
- **--llm-provider** (string) - Optional - LLM provider (e.g., ollama).
- **--llm-name** (string) - Optional - Name of the LLM to use (e.g., llama3.2).
- **--project-dir** (string) - Optional - Directory to save the updated file.

### Request Example
```bash
llmstxt-architect \
  --existing-llms-file ./llms.txt \
  --update-descriptions-only \
  --llm-provider ollama \
  --llm-name llama3.2 \
  --project-dir updated
```

### Response
#### Success Response
An updated `llms.txt` file with new descriptions, saved in the specified `project-dir`.

#### Response Example
```text
# Updated llms.txt content
```
```

--------------------------------

### Install Docusaurus llms.txt Plugin (Docusaurus)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This code shows the npm installation command for the docusaurus-plugin-llms-txt and how to include it in the docusaurus.config.js file.

```bash
npm install docusaurus-plugin-llms-txt
```

```javascript
// docusaurus.config.js
module.exports = {
  plugins: ['docusaurus-plugin-llms-txt'],
}
```

--------------------------------

### Custom Generation Prompt Example

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Shows how to override the default generation prompt using the `--summary-prompt` argument in the CLI. This allows for customized instructions to the LLM.

```bash
llmstxt-architect \
  --urls https://example.com \
  --summary-prompt "Summarize this API documentation. Focus on endpoints and use cases." \
  --project-dir output
```

--------------------------------

### Install VitePress llms Plugin (VitePress)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This snippet demonstrates how to install and configure the VitePress llms plugin. It uses npm for installation and requires adding the plugin to the Vite configuration in the .vitepress/config.js file.

```bash
npm install vitepress-plugin-llms
```

```javascript
// .vitepress/config.js
import llmsPlugin from 'vitepress-plugin-llms'

export default {
  vite: {
    plugins: [llmsPlugin()],
  },
}
```

--------------------------------

### Basic Usage with Local Model (Ollama)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example shows how to use llmstxt-architect with a local Ollama model.

```APIDOC
## Basic Usage with Local Model (Ollama)

### Description
Uses a local Ollama model to generate descriptions for provided URLs.

### Method
CLI Command

### Endpoint
N/A (CLI Tool)

### Parameters
#### Query Parameters
- **--urls** (string) - Required - URL to process
- **--max-depth** (integer) - Optional - Crawl depth (1-5), Default: 5
- **--llm-name** (string) - Optional - Model name, e.g., llama3.2:latest
- **--llm-provider** (string) - Optional - Provider (anthropic, openai, ollama), Default: anthropic
- **--project-dir** (string) - Optional - Results directory, Default: llms_txt
- **--output-file** (string) - Optional - Output file name, Default: llms.txt
- **--blacklist-file** (string) - Optional - File with URLs to exclude
- **--extractor** (string) - Optional - Extraction method (default, bs4), Default: default

### Request Example
```bash
# Start Ollama
ollama serve

llmstxt-architect \
  --urls https://docs.example.com \
  --max-depth 1 \
  --llm-name llama3.2:latest \
  --llm-provider ollama \
  --project-dir output
```

### Response
#### Success Response
Generates an `llms.txt` file in the specified `project-dir`.

#### Response Example
```text
# llms.txt content will be generated here
```
```

--------------------------------

### Raycast Extension for llms.txt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This is a command to install or link a Raycast extension for llms.txt. It assumes a Raycast extension is available at the specified URL.

```bash
# Raycast extension for llms.txt
raycast://extensions/llmstxt
```

--------------------------------

### Install Starlight llms.txt Plugin (Astro)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This snippet shows how to add the Starlight llms.txt integration to an Astro project. It requires pnpm for installation and configuration within the astro.config.mjs file.

```bash
pnpm add starlight-llms-txt
```

```javascript
// astro.config.mjs
import starlightLlmsTxt from 'starlight-llms-txt'

export default defineConfig({
  integrations: [
    starlight({
      plugins: [starlightLlmsTxt()],
    }),
  ],
})
```

--------------------------------

### Explain API Output with Claude Code

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llm-codes.md

This example shows how to pipe the output of the llm.codes scrape API directly to the 'claude' command-line tool. This allows for immediate analysis or explanation of the scraped documentation by an AI model.

```bash
# Get documentation
curl -s "https://llm.codes/api/scrape?url=https://docs.example.com" \
  | claude "Explain how to use this API"
```

--------------------------------

### Exclude URLs using Blacklist File

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example demonstrates how to exclude specific URLs from processing by providing a blacklist file.

```APIDOC
## Exclude URLs using Blacklist File

### Description
Prevents the tool from processing specified URLs by providing a file containing a list of URLs to exclude.

### Method
CLI Command

### Endpoint
N/A (CLI Tool)

### Parameters
#### Query Parameters
- **--urls** (string) - Required - The starting URL(s) for processing.
- **--blacklist-file** (string) - Required - Path to the file containing URLs to exclude.
- **--project-dir** (string) - Optional - Directory to save results.

### Request Example
```bash
# Create blacklist.txt with content like:
# https://example.com/old-api/
# https://example.com/v1/

llmstxt-architect \
  --urls https://example.com \
  --blacklist-file blacklist.txt \
  --project-dir output
```

### Response
#### Success Response
Generates an `llms.txt` file in the specified `project-dir`, excluding content from blacklisted URLs.

#### Response Example
```text
# llms.txt content excluding blacklisted URLs
```
```

--------------------------------

### Python API Usage

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example shows how to use the `generate_llms_txt` function from the llmstxt-architect Python library.

```APIDOC
## Python API Usage

### Description
Programmatically generate `llms.txt` using the `generate_llms_txt` function from the library.

### Method
Python Function Call

### Endpoint
N/A (Python API)

### Parameters
- **urls** (list[str]) - Required - List of URLs to process.
- **max_depth** (int) - Optional - Crawl depth (1-5), Default: 5.
- **llm_name** (str) - Optional - LLM model name, Default: 'claude-3-sonnet'.
- **llm_provider** (str) - Optional - LLM provider ('anthropic', 'openai', 'ollama'), Default: 'anthropic'.
- **project_dir** (str) - Optional - Directory to save results, Default: 'llms_txt'.
- **extractor** (callable) - Optional - Custom function for content extraction.

### Request Example
```python
import asyncio
from llmstxt_architect.main import generate_llms_txt

async def main():
    await generate_llms_txt(
        urls=["https://docs.example.com"],
        max_depth=2,
        llm_name="claude-3-5-sonnet-latest",
        llm_provider="anthropic",
        project_dir="output",
    )

asyncio.run(main())
```

### Response
#### Success Response
Generates an `llms.txt` file in the specified `project_dir`.

#### Response Example
```text
# llms.txt content will be generated here
```
```

--------------------------------

### Custom Summary Prompt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example shows how to override the default LLM prompt used for generating summaries.

```APIDOC
## Custom Summary Prompt

### Description
Allows customization of the prompt sent to the LLM for generating page summaries, influencing the output focus.

### Method
CLI Command

### Endpoint
N/A (CLI Tool)

### Parameters
#### Query Parameters
- **--urls** (string) - Required - URL to process.
- **--summary-prompt** (string) - Required - The custom prompt to use for summarization.
- **--project-dir** (string) - Optional - Directory to save results.

### Request Example
```bash
llmstxt-architect \
  --urls https://example.com \
  --summary-prompt "Summarize this API documentation. Focus on endpoints and use cases." \
  --project-dir output
```

### Response
#### Success Response
Generates an `llms.txt` file with summaries created using the custom prompt.

#### Response Example
```text
# llms.txt content with custom summaries
```
```

--------------------------------

### Markdown File Structure Example

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/specification/format.md

Demonstrates the basic structure of an `llms.txt` file, including required H1 title, recommended blockquote description, optional sections, and link formatting.

```markdown
# H1 Title (required)

> Blockquote with brief description (recommended)

Additional description text (optional)

## H2 Section (optional)

- [Link Name](URL): Resource description
- [Another Link](URL): What this resource contains

## Optional

- [Additional Resource](URL): Can be skipped when context is limited
```

--------------------------------

### Basic Usage with Anthropic Claude

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example shows how to use llmstxt-architect with Anthropic Claude to generate descriptions for provided URLs.

```APIDOC
## Basic Usage with Anthropic Claude

### Description
Uses Anthropic Claude to generate descriptions for provided URLs.

### Method
CLI Command

### Endpoint
N/A (CLI Tool)

### Parameters
#### Query Parameters
- **--urls** (string) - Required - URL to process
- **--max-depth** (integer) - Optional - Crawl depth (1-5), Default: 5
- **--llm-name** (string) - Optional - Model name, Default: claude-3-sonnet
- **--llm-provider** (string) - Optional - Provider (anthropic, openai, ollama), Default: anthropic
- **--project-dir** (string) - Optional - Results directory, Default: llms_txt
- **--output-file** (string) - Optional - Output file name, Default: llms.txt
- **--blacklist-file** (string) - Optional - File with URLs to exclude
- **--extractor** (string) - Optional - Extraction method (default, bs4), Default: default

### Request Example
```bash
export ANTHROPIC_API_KEY=sk-ant-...

llmstxt-architect \
  --urls https://docs.example.com \
  --max-depth 2 \
  --llm-name claude-3-5-sonnet-latest \
  --llm-provider anthropic \
  --project-dir output
```

### Response
#### Success Response
Generates an `llms.txt` file in the specified `project-dir`.

#### Response Example
```text
# llms.txt content will be generated here
```
```

--------------------------------

### Update Descriptions Only from Local llms.txt with Ollama

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Example of updating descriptions in an existing llms.txt file using a local Ollama model. Reads from a local file path.

```bash
llmstxt-architect \
  --existing-llms-file ./llms.txt \
  --update-descriptions-only \
  --llm-provider ollama \
  --llm-name llama3.2 \
  --project-dir updated
```

--------------------------------

### Custom Extractor Function

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This example shows how to define and use a custom content extractor function with the Python API.

```APIDOC
## Custom Extractor Function

### Description
Defines and utilizes a custom function to extract specific content from HTML, bypassing default extraction logic.

### Method
Python Function Definition and Usage

### Endpoint
N/A (Python API)

### Parameters
- **extractor** (callable) - Required - A function that takes HTML string and returns extracted text.

### Request Example
```python
from bs4 import BeautifulSoup
import asyncio
from llmstxt_architect.main import generate_llms_txt

def custom_extractor(html: str) -> str:
    soup = BeautifulSoup(html, 'html.parser')

    # Remove navigation and footer
    for tag in soup.find_all(['nav', 'footer', 'aside']):
        tag.decompose()

    # Extract only main content
    main = soup.find('main') or soup.find('article')
    if main:
        return main.get_text(separator='\n', strip=True)

    return soup.get_text(separator='\n', strip=True)

async def main():
    await generate_llms_txt(
        urls=["https://docs.example.com"],
        extractor=custom_extractor,
        project_dir="output"
    )

asyncio.run(main())
```

### Response
#### Success Response
Generates an `llms.txt` file using the custom extraction logic.

#### Response Example
```text
# llms.txt content extracted using the custom function
```
```

--------------------------------

### Parse llms.txt in PHP

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This PHP snippet demonstrates how to install the llmstxt/parser via composer and use the Parser class to parse the content of an llms.txt file.

```bash
composer require llmstxt/parser

```

```php
use LlmsTxt\Parser;

$parser = new Parser();
$data = $parser->parse(file_get_contents('llms.txt'));
```

--------------------------------

### Parse llms.txt in Python

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This Python snippet demonstrates how to install the 'llms-txt' library using pip, parse an llms.txt file, and convert its content into a context format. It requires reading the file content first.

```bash
pip install llms-txt
```

```python
from llms_txt import parse_llms_txt, to_context

# Parsing
with open('llms.txt') as f:
    data = parse_llms_txt(f.read())

# Convert to context
context = to_context(data, include_optional=False)
```

--------------------------------

### Markdown: Minimal Project Documentation

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/examples/basic.md

Demonstrates the absolute minimum structure for a project's README file, consisting only of a title. It also shows a recommended structure including a brief description and a link to external documentation.

```markdown
# My Project
```

```markdown
# My Project

> Brief description of what this project does.

## Documentation

- [Docs](https://example.com/docs): Main documentation
```

--------------------------------

### Project Dependencies and Build Scripts (package.json)

Source: https://context7.com/kaktaknet/llms-full-txt.ru/llms.txt

This JSON object defines the project's dependencies, including Astro, Starlight, and various utility libraries. It also specifies scripts for development, building the project with a post-build step, and previewing the site.

```json
{
  "name": "llms-full-txt",
  "type": "module",
  "version": "0.0.1",
  "scripts": {
    "dev": "astro dev",
    "build": "astro build && node scripts/postbuild.mjs",
    "preview": "astro preview",
    "astro": "astro"
  },
  "dependencies": {
    "@astrojs/starlight": "^0.37.0",
    "@fontsource/geist-mono": "^5.0.0",
    "@fontsource/geist-sans": "^5.0.0",
    "astro": "^5.0.0",
    "marked": "^17.0.1",
    "sharp": "^0.33.0",
    "starlight-llms-txt": "^0.6.0",
    "starlight-theme-black": "^0.4.0",
    "web-vitals": "^5.1.0"
  },
  "devDependencies": {
    "@astrojs/check": "^0.9.6",
    "typescript": "^5.9.3"
  }
}
```

--------------------------------

### Basic Usage: Generate llms.txt with Local Ollama Model

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Illustrates generating descriptions using a locally hosted LLM via Ollama. Ensure Ollama is running before execution.

```bash
# Start Ollama
ollama serve

llmstxt-architect \
  --urls https://docs.example.com \
  --max-depth 1 \
  --llm-name llama3.2:latest \
  --llm-provider ollama \
  --project-dir output
```

--------------------------------

### Generate llms.txt and llms-full.txt with GitHub Actions

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/examples/advanced.md

This GitHub Actions workflow automates the generation of 'llms.txt' and 'llms-full.txt' files. It runs on pushes to the 'docs' directory and weekly schedules, using 'llmstxt' CLI tool to fetch and process sitemap data.

```yaml
name: Update llms.txt

on:
  push:
    paths:
      - 'docs/**'
  schedule:
    - cron: '0 0 * * 0'  # Weekly

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Generate llms.txt
        run: |
          npx llmstxt gen https://docs.example.com/sitemap.xml \
            -ep "**/blog/**" \
            -t "My Docs"
            > public/llms.txt

      - name: Generate llms-full.txt
        run: |
          npx llmstxt gen-full https://docs.example.com/sitemap.xml \
            -ep "**/blog/**"
            > public/llms-full.txt

      - name: Commit and push
        run: |
          git config user.name "GitHub Action"
          git config user.email "action@github.com"
          git add public/llms.txt public/llms-full.txt
          git diff --staged --quiet || git commit -m "Update llms.txt"
          git push
```

--------------------------------

### Basic Usage: Generate llms.txt with OpenAI

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Shows how to generate descriptions using OpenAI's models via the CLI. Requires setting the OPENAI_API_KEY environment variable.

```bash
export OPENAI_API_KEY=sk-...

llmstxt-architect \
  --urls https://docs.example.com \
  --max-depth 2 \
  --llm-name gpt-4o \
  --llm-provider openai \
  --project-dir output
```

--------------------------------

### Basic llms.txt Generation with llmstxt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Shows the basic command to generate a 'llms.txt' file by providing a sitemap URL. The output is redirected to 'llms.txt'. This command generates a list of pages and their descriptions.

```bash
npx llmstxt gen https://example.com/sitemap.xml > llms.txt
```

--------------------------------

### Generate llms.txt from Sitemap using CLI (Bash)

Source: https://context7.com/kaktaknet/llms-full-txt.ru/llms.txt

This Bash script demonstrates how to use the llmstxt CLI tool to generate llms.txt files from a website's sitemap. It covers basic generation, full version generation, and advanced options like filtering paths, replacing titles, and setting metadata. It also shows short-form options.

```bash
# Install llmstxt CLI globally
npm install -g llmstxt

# Generate basic llms.txt with links and descriptions
npx llmstxt gen https://example.com/sitemap.xml > llms.txt

# Generate full version with complete page content
npx llmstxt gen-full https://example.com/sitemap.xml > llms-full.txt

# Advanced: Filter paths and customize metadata
npx llmstxt gen https://docs.example.com/sitemap.xml \
  --exclude-path "**/blog/**" \
  --exclude-path "**/changelog/**" \
  --exclude-path "**/privacy**" \
  --exclude-path "**/terms**" \
  --replace-title 's/\| Docs//' \
  --title 'Example Docs' \
  --description 'Official documentation for Example' \
  --concurrency 5 \
  > llms.txt

# Short form options
npx llmstxt gen https://example.com/sitemap.xml \
  -ep "**/blog/**" \
  -rt 's/\| Example//' \
  -t "My Documentation" \
  -d "Project documentation" \
  -c 3 \
  > llms.txt
```

--------------------------------

### Resuming Interrupted Process

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Illustrates how to resume a process that was interrupted. The tool automatically skips already processed URLs based on the project directory's checkpoints.

```bash
# If the process was interrupted, just run again
llmstxt-architect --urls https://example.com --project-dir output

# Processed URLs will be skipped
```

--------------------------------

### Custom Metadata with llmstxt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Shows how to set a custom title and description for the generated 'llms.txt' file using the `--title` and `--description` options. This is useful for branding or specific project needs.

```bash
npx llmstxt gen https://example.com/sitemap.xml \
  --title "My Documentation" \
  --description "Official docs for My Project"
```

--------------------------------

### Python API: Custom HTML Extractor Function

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Provides an example of defining a custom HTML extractor function for the Python API. This function uses BeautifulSoup to parse HTML and extract relevant content, excluding navigation and footers.

```python
from bs4 import BeautifulSoup

def custom_extractor(html: str) -> str:
    soup = BeautifulSoup(html, 'html.parser')

    # Remove navigation and footer
    for tag in soup.find_all(['nav', 'footer', 'aside']):
        tag.decompose()

    # Extract only main content
    main = soup.find('main') or soup.find('article')
    if main:
        return main.get_text(separator='\n', strip=True)

    return soup.get_text(separator='\n', strip=True)

# Example usage within generate_llms_txt:
# await generate_llms_txt(
#     urls=["https://docs.example.com"],
#     extractor=custom_extractor,
#     # ...
# )
```

--------------------------------

### Checkpoints and Resumption

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This explains how the tool automatically saves progress and can be resumed after interruptions.

```APIDOC
## Checkpoints and Resumption

### Description
The tool automatically saves progress (checkpoints) periodically. If a process is interrupted, running the same command again will resume from the last saved checkpoint, skipping already processed URLs.

### Method
CLI Command (Automatic)

### Endpoint
N/A (CLI Tool)

### Parameters
#### Query Parameters
- **--urls** (string) - Required - The starting URL(s) for processing. The tool will automatically detect existing checkpoints.
- **--project-dir** (string) - Required - The directory where checkpoints are stored and results are saved.

### Request Example
```bash
# If the process was interrupted, just run again with the same parameters:
llmstxt-architect --urls https://example.com --project-dir output

# Processed URLs will be skipped automatically.
```

### Response
#### Success Response
Resumes processing from the last checkpoint or completes the run if no interruption occurred.

#### Response Example
```text
# Resumed processing or completed run.
```
```

--------------------------------

### Basic Usage: Generate llms.txt with Anthropic Claude

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Demonstrates how to use the llmstxt-architect CLI to generate descriptions using Anthropic's Claude model. Requires setting the ANTHROPIC_API_KEY environment variable.

```bash
export ANTHROPIC_API_KEY=sk-ant-...

llmstxt-architect \
  --urls https://docs.example.com \
  --max-depth 2 \
  --llm-name claude-3-5-sonnet-latest \
  --llm-provider anthropic \
  --project-dir output
```

--------------------------------

### Generate LLM Text Files with GitHub Actions

Source: https://context7.com/kaktaknet/llms-full-txt.ru/llms.txt

This workflow automates the generation of 'llms.txt' and 'llms-full.txt' files. It checks out the repository, sets up Node.js, runs the 'llmstxt' CLI tool with specified configurations to fetch and process data from a sitemap, and commits any changes back to the repository. The process is scheduled to run weekly.

```yaml
name: Generate llms.txt

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 0 * * 0'  # Every Sunday at midnight

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Generate llms.txt
        run: |
          npx llmstxt gen https://docs.example.com/sitemap.xml \
            -ep "**/blog/**" \
            -ep "**/privacy**" \
            -rt 's/| Docs//' \
            > public/llms.txt

      - name: Generate llms-full.txt
        run: |
          npx llmstxt gen-full https://docs.example.com/sitemap.xml \
            -ep "**/blog/**" \
            > public/llms-full.txt

      - name: Commit and push if changed
        run: |
          git config --local user.email "action@github.com"
          git config --local user.name "GitHub Action"
          git add public/llms.txt public/llms-full.txt
          git diff --quiet && git diff --staged --quiet || \
            (git commit -m "Update llms.txt files" && git push)
```

--------------------------------

### GitHub Actions for llms.txt Generation

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

A YAML configuration for GitHub Actions to automate the generation and commit of 'llms.txt' files. This workflow runs on push to the main branch and weekly schedules.

```yaml
name: Generate llms.txt

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 0 * * 0'  # Every Sunday

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Generate llms.txt
        run: |
          npx llmstxt gen https://docs.example.com/sitemap.xml \
            -ep "**/blog/**" > public/llms.txt

      - name: Commit changes
        run: |
          git config --local user.email "action@github.com"
          git config --local user.name "GitHub Action"
          git add public/llms.txt
          git commit -m "Update llms.txt" || exit 0
          git push
```

--------------------------------

### Exclude URLs using a Blacklist File

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Demonstrates how to prevent the tool from crawling specific URLs by providing a blacklist file. The file contains one URL per line.

```text
# Deprecated pages
https://example.com/old-api/
https://example.com/v1/

# Not relevant
https://example.com/jobs/
https://example.com/legal/
```

```bash
llmstxt-architect \
  --urls https://example.com \
  --blacklist-file blacklist.txt \
  --project-dir output
```

--------------------------------

### Self-Host Environment Variables

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llm-codes.md

This lists the essential environment variables required for self-hosting the llm.codes project. `FIRECRAWL_API_KEY` is mandatory for the service to function, while `UPSTASH_REDIS_URL` is optional and used for caching.

```bash
FIRECRAWL_API_KEY=fc-...  # Required
UPSTASH_REDIS_URL=...     # Optional (for cache)
```

--------------------------------

### Troubleshooting Slow Generation with llmstxt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Provides a solution for slow 'llms.txt' generation by reducing the concurrency level to avoid rate limiting issues.

```bash
npx llmstxt gen https://example.com/sitemap.xml -c 2
```

--------------------------------

### MCP Server Configuration for llms.txt Documentation

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This JSON configuration defines an MCP server named 'docs' that uses 'uvx' to process llms.txt documentation from a specified URL. The output is transported via stdio.

```json
{
  "mcpServers": {
    "docs": {
      "command": "uvx",
      "args": [
        "--from", "mcpdoc",
        "mcpdoc",
        "--urls", "LangGraph:https://langchain-ai.github.io/langgraph/llms.txt",
        "--transport", "stdio"
      ]
    }
  }
}
```

--------------------------------

### Concurrency Control with llmstxt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Explains how to control the number of concurrent requests made by llmstxt to fetch page content using the `--concurrency` option. This is important for managing server load and avoiding rate limits.

```bash
# Limit number of concurrent requests
npx llmstxt gen https://example.com/sitemap.xml \
  --concurrency 3
```

--------------------------------

### Full Content Generation with llmstxt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Command to generate a 'llms-full.txt' file containing the full Markdown content of all pages listed in the sitemap. This includes a Table of Contents and last updated dates.

```bash
npx llmstxt gen-full https://example.com/sitemap.xml > llms-full.txt
```

--------------------------------

### Python API: Basic llms.txt Generation

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

Shows the basic usage of the llmstxt_architect Python API to generate llms.txt programmatically. This asynchronous function requires an asyncio event loop.

```python
import asyncio
from llmstxt_architect.main import generate_llms_txt

async def main():
    await generate_llms_txt(
        urls=["https://docs.example.com"],
        max_depth=2,
        llm_name="claude-3-5-sonnet-latest",
        llm_provider="anthropic",
        project_dir="output",
    )

asyncio.run(main())
```

--------------------------------

### Path Filtering with llmstxt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Illustrates how to use `--exclude-path` and `--include-path` options to filter which URLs from the sitemap are processed. This allows for precise control over the content included in the generated 'llms.txt'.

```bash
# Exclude paths
npx llmstxt gen https://example.com/sitemap.xml \
  --exclude-path "**/blog/**" \
  --exclude-path "**/privacy**" \
  --exclude-path "**/terms**"

# Include only specific paths
npx llmstxt gen https://example.com/sitemap.xml \
  --include-path "**/docs/**" \
  --include-path "**/api/**"
```

--------------------------------

### MCP Server Configuration for Multiple llms.txt Sources

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/reference/configuration.md

This JSON configuration sets up an MCP server named 'multi-docs' to process llms.txt files from multiple sources (React, Next.js, Tailwind). It uses 'uvx' for processing and 'stdio' for transport.

```json
{
  "mcpServers": {
    "multi-docs": {
      "command": "uvx",
      "args": [
        "--from", "mcpdoc",
        "mcpdoc",
        "--urls",
        "React:https://react.dev/llms.txt",
        "Next.js:https://nextjs.org/llms.txt",
        "Tailwind:https://tailwindcss.com/llms.txt",
        "--transport", "stdio"
      ]
    }
  }
}
```

--------------------------------

### Troubleshooting Incorrect Titles with llmstxt

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-cli.md

Offers a solution for incorrect page titles by using the `--replace-title` option with a regular expression to clean up titles.

```bash
npx llmstxt gen url -rt 's/ - My Site$//'
```

--------------------------------

### MCP Server Configuration (JSON)

Source: https://github.com/kaktaknet/llms-full-txt.ru/blob/main/src/content/docs/en/tools/llmstxt-architect.md

This JSON configuration defines how the MCP server should process the generated llms.txt file. It specifies the command, arguments, and transport method for integrating with the MCP server.

```json
{
  "mcpServers": {
    "docs": {
      "command": "uvx",
      "args": [
        "--from", "mcpdoc",
        "mcpdoc",
        "--urls", "Docs:./output/llms.txt",
        "--transport", "stdio"
      ]
    }
  }
}
```