### MCP Installation URL Generation (JavaScript)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext/index.html

Generates a URL for installing an MCP (Master Control Program) based on a given hostname. It constructs the URL using a predefined pattern that includes the MCP name derived from the hostname. This is used to guide users to the installation page.

```javascript
function getInstallMcpUrl(hostname) { const mcpName = getMcpName(hostname); return `https://installthismcp.com/${mcpName}?url=https://mcp.llmtext.com/${hostname}/mcp`; }
```

--------------------------------

### llms.txt File Structure Example (Markdown)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/drafts/llms-txt-spec-draft.md

An example of the llms.txt file format, demonstrating the required title, optional summary, additional details, and a file list section.

```markdown
# Project Name

> Brief project summary with essential context

Additional details about the project and usage guidelines.

## Core Documentation

- [Quick Start](https://example.com/quickstart.html.md): Getting started guide
- [API Reference](https://example.com/api.html.md): Complete API documentation

## Optional

- [Advanced Topics](https://example.com/advanced.html.md): In-depth coverage

```

--------------------------------

### GitHub Actions CI/CD for Documentation Extraction

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/README.md

This GitHub Actions workflow automates the extraction of documentation daily or on demand. It installs the necessary tool, extracts content, and commits changes back to the repository. Requires the PARALLEL_API_KEY secret.

```yaml
name: Extract Documentation

on:
  schedule:
    - cron: "0 0 * * *" # Daily at midnight UTC
  workflow_dispatch: # Allow manual trigger

jobs:
  extract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Extract documentation
        env:
          PARALLEL_API_KEY: ${{ secrets.PARALLEL_API_KEY }}
        run: |
          npm install -g extract-from-sitemap
          npx extract-from-sitemap

      - name: Commit changes
        run: |
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git config user.name "github-actions[bot]"
          git add .
          git diff --quiet && git diff --staged --quiet || \
            (git commit -m "Update docs [skip ci]" && git push)
```

--------------------------------

### GitHub Actions CI/CD for llms.txt Generation

Source: https://context7.com/janwilmake/llmtext-mcp/llms.txt

This workflow automates the generation and deployment of llms.txt. It checks out code, sets up Node.js, installs dependencies, creates a configuration file, extracts documentation content, generates an asset manifest, deploys the output to GitHub Pages, and validates the generated llms.txt file.

```yaml
name: Generate llms.txt

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: |
          npm install extract-from-sitemap
          npm install llms-txt-generate
          npm install getassetmanifest

      - name: Create configuration
        run: |
          cat > llmtext.json << 'EOF'
          {
            "title": "${{ github.repository }}",
            "description": "Documentation for ${{ github.repository }}",
            "outDir": "./docs-output",
            "sources": [
              {
                "title": "Main Documentation",
                "origin": "https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}",
                "outDir": "./docs-output/main"
              }
            ]
          }
          EOF

      - name: Extract documentation
        env:
          PARALLEL_API_KEY: ${{ secrets.PARALLEL_API_KEY }}
        run: npx extract-from-sitemap

      - name: Generate asset manifest (if using static files)
        run: npx getassetmanifest

      - name: Deploy llms.txt
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./docs-output
          destination_dir: docs

      - name: Validate llms.txt
        run: |
          sleep 10  # Wait for deployment
          curl "https://check.llmtext.com/check?url=https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/docs/llms.txt" \
            | jq '.valid'

```

--------------------------------

### GitLab CI for Documentation Extraction

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/README.md

This GitLab CI configuration sets up a pipeline to extract documentation. It uses a Node.js Docker image, installs the extraction tool, runs the extraction process, and commits the updated documentation. Requires PARALLEL_API_KEY as a CI/CD variable.

```yaml
extract-docs:
  image: node:20
  script:
    - npm install -g extract-from-sitemap
    - npx extract-from-sitemap
    - |
      git config user.email "gitlab-ci@gitlab.com"
      git config user.name "GitLab CI"
      git add docs/
      git diff --quiet && git diff --staged --quiet || \
        (git commit -m "Update docs [skip ci]" && git push https://oauth2:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git HEAD:${CI_COMMIT_REF_NAME})
  only:
    - schedules
    - web
```

--------------------------------

### Install parse-llms-txt via npm

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/parse-llms-txt/README.md

Installs the parse-llms-txt package using npm. This is the first step to using the library in your Node.js or browser projects.

```bash
npm install parse-llms-txt
```

--------------------------------

### NPM Script for Sitemap Extraction

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/README.md

This command installs the 'extract-from-sitemap' package globally and then executes it to generate markdown bundles. It's typically added to `package.json` scripts for easy execution.

```bash
npm install -g extract-from-sitemap
npx extract-from-sitemap
```

--------------------------------

### Fetch Webpage Content as Markdown using JavaScript

Source: https://context7.com/janwilmake/llmtext-mcp/llms.txt

Provides a JavaScript example for fetching Markdown content from a webpage using the llmtext.reader service. It includes URL encoding and demonstrates handling the text response. Dependencies include the fetch API.

```javascript
const url = "https://blog.example.com/article";
const encodedUrl = encodeURIComponent(url).replace(/^https?:/+/, "");

// Fetch as markdown
const mdResponse = await fetch(`https://reader.llmtext.com/md/${encodedUrl}`);
const markdown = await mdResponse.text();
console.log(markdown);
```

--------------------------------

### GET /leaderboard[/{hostname}]

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext.mcp/SPEC.md

Fetches a public leaderboard showing top users and MCP servers by usage count. Results are cached for 1 hour.

```APIDOC
## GET /leaderboard[/{hostname}]

### Description
Provides a public leaderboard displaying the top users (by total count) and top MCP servers (by total usage count). Results are cached for a maximum of 1 hour.

### Method
GET

### Endpoint
`/leaderboard` or `/leaderboard/{hostname}`

### Parameters
#### Path Parameters
- **hostname** (string) - Optional - Filters the leaderboard to a specific hostname.

#### Query Parameters
None

### Request Example
`GET /leaderboard`
`GET /leaderboard/example.com`

### Response
#### Success Response (200)
- **leaderboard** (object) - Contains leaderboard information:
  - **top_users** (array) - An array of users ranked by their total usage count:
    - **username** (string) - The user's identifier (e.g., X/Twitter username).
    - **total_count** (integer) - The total usage count for the user.
  - **top_mcp_servers** (array) - An array of MCP servers ranked by their total usage count:
    - **server_hostname** (string) - The hostname of the MCP server.
    - **total_usage_count** (integer) - The total usage count across all users for this server.

#### Response Example
```json
{
  "leaderboard": {
    "top_users": [
      {
        "username": "user123",
        "total_count": 1500
      },
      {
        "username": "ai_enthusiast",
        "total_count": 1200
      }
    ],
    "top_mcp_servers": [
      {
        "server_hostname": "mcp.example.com",
        "total_usage_count": 5000
      },
      {
        "server_hostname": "another-mcp.ai",
        "total_usage_count": 3000
      }
    ]
  }
}
```
```

--------------------------------

### Example llms.txt to JSON Conversion

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/parse-llms-txt/README.md

Provides a concrete example of an llms.txt markdown content and its corresponding JSON output after being parsed by the `parse-llms-txt` tool. This helps visualize the transformation process.

```markdown
# FastHTML

> FastHTML is a python library for creating server-rendered hypermedia applications.

Important notes about compatibility and usage.

## Documentation

- [Quick Start](https://fastht.ml/docs/quickstart.html.md): A brief overview
- [HTMX Reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): HTMX documentation

## Optional

- [Full Documentation](https://fastht.ml/docs/full.md): Complete documentation
```

```json
{
  "title": "FastHTML",
  "description": "FastHTML is a python library for creating server-rendered hypermedia applications.",
  "details": "Important notes about compatibility and usage.",
  "sections": [
    {
      "name": "Documentation",
      "files": [
        {
          "name": "Quick Start",
          "url": "https://fastht.ml/docs/quickstart.html.md",
          "notes": "A brief overview"
        },
        {
          "name": "HTMX Reference",
          "url": "https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md",
          "notes": "HTMX documentation"
        }
      ]
    },
    {
      "name": "Optional",
      "files": [
        {
          "name": "Full Documentation",
          "url": "https://fastht.ml/docs/full.md",
          "notes": "Complete documentation"
        }
      ]
    }
  ]
}
```

--------------------------------

### HTML Structure and JavaScript for Check Tab Functionality

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext/CONTEXT.md

This HTML and JavaScript code provides a complete implementation for the index.html file. It includes the necessary structure for the 'check' tab, handles URL parameter parsing to trigger immediate API checks, enhances form submission to use the 'check' URL, and dynamically renders API response data in a user-friendly table format with actionable suggestions based on provided recommendations. The 'Install this MCP' button's visibility and state are managed according to the API check results.

```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>LLM Text MCP Checker</title>
    <style>
        body { font-family: sans-serif; margin: 20px; }
        table { border-collapse: collapse; width: 100%; margin-top: 20px; }
        th, td { border: 1px solid #ddd; padding: 8px; text-align: left; }
        th { background-color: #f2f2f2; }
        .recommendation { color: orange; font-weight: bold; }
        .success { color: green; font-weight: bold; }
        .error { color: red; font-weight: bold; }
        .disabled-button {
            background-color: #cccccc;
            color: #666666;
            cursor: not-allowed;
            padding: 10px 15px;
            border: none;
            border-radius: 5px;
        }
        .enabled-button {
            background-color: #4CAF50;
            color: white;
            padding: 10px 15px;
            border: none;
            border-radius: 5px;
            cursor: pointer;
        }
        .enabled-button:hover { background-color: #45a049; }
    </style>
</head>
<body>

    <h1>LLM Text MCP Checker</h1>

    <div>
        <label for="urlInput">Enter URL:</label>
        <input type="text" id="urlInput" name="urlInput" size="50">
        <button id="checkButton">Check URL</button>
    </div>

    <div id="resultsContainer" style="margin-top: 30px;">
        <h2>Check Results</h2>
        <div id="resultsTable"></div>
        <button id="installButton" class="disabled-button" disabled>Install this MCP</button>
    </div>

    <script>
        const urlInput = document.getElementById('urlInput');
        const checkButton = document.getElementById('checkButton');
        const resultsTableDiv = document.getElementById('resultsTable');
        const installButton = document.getElementById('installButton');

        const API_URL = 'https://check.llmtext.com/openapi.json'; // Placeholder, use actual API endpoint

        // Function to fetch and display check results
        async function performCheck(url) {
            resultsTableDiv.innerHTML = 'Loading...';
            installButton.className = 'disabled-button';
            installButton.disabled = true;

            try {
                // In a real scenario, you would call the actual API endpoint
                // For demonstration, we simulate an API response
                const response = await fetch(API_URL + '?url=' + encodeURIComponent(url));
                const data = await response.json(); // Assuming the API returns JSON

                // Simulate response processing and rendering
                renderResults(data);

            } catch (error) {
                console.error('Error during check:', error);
                resultsTableDiv.innerHTML = '<p class="error">Failed to perform check. Please try again later.</p>';
                installButton.className = 'disabled-button';
                installButton.disabled = true;
            }
        }

        // Function to render results in a table
        function renderResults(data) {
            let tableHTML = '<table><thead><tr><th>Status</th><th>Details</th><th>Suggestions</th></tr></thead><tbody>';
            let allGood = true;

            if (data && data.issues && data.issues.length > 0) {
                data.issues.forEach(issue => {
                    allGood = false;
                    const statusClass = issue.severity === 'error' ? 'error' : (issue.severity === 'warning' ? 'recommendation' : 'success');
                    tableHTML += `<tr>
                                    <td class="${statusClass}">${issue.severity.toUpperCase()}</td>
                                    <td>${issue.message}</td>
                                    <td>${issue.suggestions ? issue.suggestions.join(', ') : 'N/A'}</td>
                                </tr>`;
                });
            } else {
                tableHTML += '<tr><td colspan="3" class="success">All checks passed!</td></tr>';
            }

            tableHTML += '</tbody></table>';
            resultsTableDiv.innerHTML = tableHTML;

            if (allGood) {
                installButton.className = 'enabled-button';
                installButton.disabled = false;
            }
        }

        // Handle check button click
        checkButton.addEventListener('click', () => {
            const url = urlInput.value.trim();
            if (url) {
                performCheck(url);
            } else {
                alert('Please enter a URL to check.');
            }
        });

        // Handle form submission to link to ?check={url}
        document.getElementById('searchForm').addEventListener('submit', function(event) {
            event.preventDefault(); // Prevent default form submission
            const url = urlInput.value.trim();
            if (url) {
                window.location.href = `?check=${encodeURIComponent(url)}`;
            } else {
                alert('Please enter a URL to check.');
            }
        });

        // Initialize check if URL parameter is present
        window.addEventListener('load', () => {
            const params = new URLSearchParams(window.location.search);
            const checkUrl = params.get('check');
            if (checkUrl) {
                urlInput.value = checkUrl;
                performCheck(checkUrl);
            }
        });
    </script>

</body>
</html>

```

--------------------------------

### Enhanced llms.txt Implementation with Subpath Linking (HTTP)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/drafts/SEP-001-discovery.md

This example illustrates an enhanced HTTP interaction where a request for an HTML page includes a 'Link' header pointing to a context-specific llms.txt file located in a subpath. This showcases the preferred method for linking to secondary llms.txt locations.

```http
GET /docs/api/endpoint-guide.html HTTP/1.1
Host: example.com

HTTP/1.1 200 OK
Content-Type: text/html
Link: </docs/api/llms.txt>; rel="llms-txt"
```

--------------------------------

### GET /get/{url}

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext.mcp/SPEC.md

Fetches the text content from a given URL. It stores fetch history and provides warnings for non-plaintext or overly large responses.

```APIDOC
## GET /get/{url}

### Description
Fetches the text content from a given URL. This endpoint is designed to retrieve plaintext content and includes safeguards against non-plaintext or excessively large responses.

### Method
GET

### Endpoint
`/get/{url}`

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL from which to fetch content.

#### Query Parameters
None

### Request Example
`GET /get/https://example.com/document.txt`

### Response
#### Success Response (200)
- **content** (string) - The text content fetched from the URL.
- **message** (string) - Informational message, e.g., warning about content type or size.

#### Response Example
```json
{
  "content": "This is the text content from the URL.",
  "message": "URL content fetched successfully."
}
```

#### Error Response (400)
- **error** (string) - Description of the error (e.g., "Invalid URL", "Content is not plaintext", "Content too large").

#### Response Example
```json
{
  "error": "Content is not plaintext"
}
```
```

--------------------------------

### URL and Domain Utility Functions (JavaScript)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext/index.html

Provides utility functions for web development, including extracting the apex domain from a given domain, generating favicon URLs, formatting domain names for MCP installation, and extracting hostnames from URLs. These functions are crucial for dynamic URL manipulation and asset retrieval.

```javascript
function extractApexDomain(domain) { const parts = domain.split('.'); if (parts.length < 2) return domain; return parts.slice(-2).join('.'); }
function getFaviconUrl(domain) { return `https://www.google.com/s2/favicons?domain=${extractApexDomain(domain)}&sz=32`; }
function getMcpName(domain) { return domain.replace(/\./g, '-'); }
function extractHostname(url) { try { const urlObj = new URL(url); return urlObj.hostname; } catch (e) { return null; } }
```

--------------------------------

### Rendering Validation Results (JavaScript)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext/index.html

Renders the validation results in the UI, dynamically creating HTML based on the validation status (`valid` or `invalid`), errors, warnings, and metadata received from the API. It also provides a suggestion to recreate the llms.txt file and a button to install the MCP if validation passes.

```javascript
function renderValidationResult(data, url) { const resultDiv = document.getElementById('validation-result'); const validClass = data.valid ? 'valid' : 'invalid'; const icon = data.valid ? '✓' : '✗'; const title = data.valid ? 'VALID llms.txt' : 'INVALID llms.txt'; const hostname = extractHostname(url); const installUrl = hostname ? getInstallMcpUrl(hostname) : '#'; const installDisabled = !data.valid ? 'disabled' : ''; let html = `<div class="validation-result ${validClass}"><h4>${icon} ${title}</h4>`; if (data.errors && data.errors.length > 0) { html += '<h5 style="margin-top: 1rem; margin-bottom: 0.5rem; font-size: 0.9rem;">ERRORS:</h5>'; html += '<ul class="validation-list errors">'; data.errors.forEach(error => { html += `<li>${error}</li>`; }); html += '</ul>'; } if (data.warnings && data.warnings.length > 0) { html += '<h5 style="margin-top: 1rem; margin-bottom: 0.5rem; font-size: 0.9rem;">WARNINGS:</h5>'; html += '<ul class="validation-list warnings">'; data.warnings.forEach(warning => { html += `<li>${warning}</li>`; }); html += '</ul>'; } if (data.metadata) { html += '<div class="metadata-section"><h5>METADATA</h5><table class="metadata-table">'; for (const [key, value] of Object.entries(data.metadata)) { if (value !== null && value !== undefined) { const displayValue = typeof value === 'object' ? JSON.stringify(value, null, 2) : value; html += `<tr><td>${key}</td><td>${displayValue}</td></tr>`; } } html += '</table></div>'; } if (!data.valid) { html += ` <div class="fix-suggestion"> <h5>💡 SUGGESTION</h5> <p>Fix your llms.txt by recreating it with our library.</p> <a href="#" onclick="event.preventDefault(); switchTab(0);">RECREATE WITH LIBRARY</a> </div> `; } html += `<a href="${installUrl}" class="install-mcp-btn ${installDisabled}" ${data.valid ? 'target="_blank" rel="noopener"' : ''}>INSTALL THIS MCP</a>`; html += '</div>'; resultDiv.innerHTML = html; }
```

--------------------------------

### Extraction Form Handling in JavaScript

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/index.html

Handles the submission of the extraction form. It prevents the default form submission, retrieves the hostname, sets the loading state, and initiates the extraction process by making a GET request to the `/extract` endpoint.

```javascript
async function handleExtraction(e) {
  e.preventDefault();
  const hostname = document.getElementById('hostname').value.trim();
  if (!hostname) return;

  setLoadingState(true);
  showProgress();

  try {
    const response = await fetch(`/extract?hostname=${encodeURIComponent(hostname)}`, {
      method: 'GET',
      headers: {
        'Authorization': `B`
      }
    });
    // ... rest of the extraction logic
```

--------------------------------

### Example JSON Data Structure for User and Server Metrics

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext/index.html

This JSON object contains sample data representing user activity and server performance metrics. It includes arrays for 'users' and 'servers', with each entry detailing properties like username, request counts, token usage, and server validity. This data structure can be used to track and analyze AI traffic.

```json
{
  "users": [
    {
      "username": "janwilmake",
      "total_requests": 64,
      "total_tokens": 233133,
      "profile_image_url": null
    },
    {
      "username": "flowisgreat",
      "total_requests": 1,
      "total_tokens": 3875,
      "profile_image_url": null
    }
  ],
  "servers": [
    {
      "hostname": "parallel.ai",
      "total_requests": 52,
      "total_tokens": 148546,
      "unique_users": 2,
      "valid": true
    },
    {
      "hostname": "docs.tavus.io",
      "total_requests": 3,
      "total_tokens": 5211,
      "unique_users": 1,
      "valid": true
    },
    {
      "hostname": "llmtext.com",
      "total_requests": 3,
      "total_tokens": 21898,
      "unique_users": 1,
      "valid": true
    },
    {
      "hostname": "docs.cursor.com",
      "total_requests": 2,
      "total_tokens": 35928,
      "unique_users": 1,
      "valid": true
    },
    {
      "hostname": "docs.perplexity.ai",
      "total_requests": 2,
      "total_tokens": 8785,
      "unique_users": 1,
      "valid": true
    },
    {
      "hostname": "bun.com",
      "total_requests": 1,
      "total_tokens": 2204,
      "unique_users": 1,
      "valid": true
    },
    {
      "hostname": "docs.x.com",
      "total_requests": 1,
      "total_tokens": 13014,
      "unique_users": 1,
      "valid": true
    },
    {
      "hostname": "vuejs.org",
      "total_requests": 1,
      "total_tokens": 1422,
      "unique_users": 1,
      "valid": true
    },
    {
      "valid": true,
      "hostname": "developers.cloudflare.com",
      "rank": 5,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "ast-grep.github.io",
      "rank": 123,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "community-charts.github.io",
      "rank": 123,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.sourceforge.net",
      "rank": 212,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "weather.com",
      "rank": 224,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "developers.hubspot.com",
      "rank": 246,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "docs.linktr.ee",
      "rank": 249,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "docs.docker.com",
      "rank": 551,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.kick.com",
      "rank": 907,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.cdp.coinbase.com",
      "rank": 1791,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "equipment-python.vercel.app",
      "rank": 2019,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "future-first-design.vercel.app",
      "rank": 2019,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "docs.sardine.ai",
      "rank": 2882,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.hybrid.ai",
      "rank": 2902,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "hola.org",
      "rank": 3037,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.anthropic.com",
      "rank": 3105,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.zapier.com",
      "rank": 3120,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": false,
      "hostname": "ajaib.gitbook.io",
      "rank": 3420,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.joinmassive.com",
      "rank": 4319,
      "total_requests": 0,
      "total_tokens": 0,
      "unique_users": 0
    },
    {
      "valid": true,
      "hostname": "docs.cod"
    }
  ]
}
```

--------------------------------

### Configure Asset Filtering (.assetsignore)

Source: https://context7.com/janwilmake/llmtext-mcp/llms.txt

Configures asset filtering for the getassetmanifest tool using a `.assetsignore` file, which follows gitignore syntax. This example demonstrates creating such a file to exclude temporary files, build artifacts, private files, development files, and specific patterns or directories. The CLI tool will then respect these patterns during manifest generation.

```bash
# Create .assetsignore file
cat > .assetsignore << 'EOF'
# Ignore temporary files
*.tmp
*.log
*.swp

# Ignore build artifacts
dist/
build/
.cache/

# Ignore private files
private/
secrets/
*.key

# Ignore node modules
node_modules/

# Ignore development files
.env
.env.*
wrangler.toml
package.json
tsconfig.json

# Ignore specific patterns
draft-*
test-*.md

# Ignore entire directories
drafts/
temp/
EOF

# Generate manifest (respects .assetsignore)
node node_modules/getassetmanifest/cli.js
```

--------------------------------

### JavaScript: Progress Display and Animation

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/index.html

Handles the display and animation of the extraction progress. It reveals a progress section, hides others, and updates a progress bar and text with simulated progress. It also starts an interval timer to animate the progress bar, which is stored globally to allow for clearing later.

```javascript
function showProgress() { progressSection.classList.remove('hidden'); resultsSection.classList.add('hidden'); errorSection.classList.add('hidden'); // Animate progress bar const progressFill = document.getElementById('progress-fill'); const progressText = document.getElementById('progress-text'); let progress = 0; const interval = setInterval(() => { progress += Math.random() * 15; if (progress > 90) progress = 90; progressFill.style.width = `${progress}%`; if (progress < 30) { progressText.textContent = 'Discovering sitemap...'; } else if (progress < 60) { progressText.textContent = 'Processing pages...'; } else { progressText.textContent = 'Generating markdown...'; } }, 300); // Store interval to clear it later window.progressInterval = interval; }
```

--------------------------------

### Run Extract-from-Sitemap with CLI

Source: https://context7.com/janwilmake/llmtext-mcp/llms.txt

This section provides command-line instructions for using the 'extract-from-sitemap' tool. It covers creating a configuration file, running the tool interactively or with an API key for CI environments, and illustrates the expected output directory structure. The commands demonstrate setting up the configuration and executing the extraction process.

```bash
# Create llmtext.json configuration
cat > llmtext.json << 'EOF'
{
  "title": "My Docs",
  "description": "Documentation site",
  "outDir": "./docs",
  "sources": [
    {
      "title": "Main Docs",
      "origin": "https://docs.example.com",
      "outDir": "./docs/main"
    }
  ]
}
EOF

# Run extraction (interactive OAuth if needed)
npx extract-from-sitemap

# Or set API key via environment variable (for CI)
export PARALLEL_API_KEY="your-api-key"
npx extract-from-sitemap

# Output structure:
# ./docs/
#   llms.txt          (generated index)
#   main/
#     /index.md       (extracted pages)
#     /api.md
#     /guide.md

# In CI environments (GitHub Actions, GitLab CI, etc.)
# The tool detects CI and requires PARALLEL_API_KEY env var

```

--------------------------------

### JavaScript: Setting Up Download and View Handlers

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/index.html

Attaches event listeners to UI buttons for initiating file downloads and viewing specific files. It sets up handlers for the 'download-btn' to trigger a 'downloadAllFilesAsZip' function and for the 'view-llms-btn' to call 'viewLlmsTxt'.

```javascript
function setupDownloadHandlers() { document.getElementById('download-btn').addEventListener('click', downloadAllFilesAsZip); document.getElementById('view-llms-btn').addEventListener('click', viewLlmsTxt); }
```

--------------------------------

### Deploy Documentation using peaceiris/actions-gh-pages

Source: https://context7.com/janwilmake/llmtext-mcp/llms.txt

This action deploys documentation content to GitHub Pages. It requires a GitHub token, specifies the source directory (`./docs-output`), and the destination directory on the GitHub Pages branch (`docs`).

```yaml
uses: peaceiris/actions-gh-pages@v3
with:
  github_token: ${{ secrets.GITHUB_TOKEN }}
  publish_dir: ./docs-output
  destination_dir: docs
```

--------------------------------

### GET /usage

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext.mcp/SPEC.md

Retrieves the user's top LLM.txt URLs and their total history count on a specific hostname. Requires authorization.

```APIDOC
## GET /usage

### Description
Returns a list of the user's most frequently accessed LLM.txt URLs and the total count of history entries for each hostname, associated with the authenticated user.

### Method
GET

### Endpoint
`/usage`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

### Request Example
`GET /usage` (Requires authentication)

### Response
#### Success Response (200)
- **usage_data** (array) - An array of objects, where each object contains:
  - **hostname** (string) - The hostname for the usage data.
  - **top_llms_txt_urls** (array) - An array of objects representing top LLM.txt URLs:
    - **url** (string) - The LLM.txt URL.
    - **count** (integer) - The total count of history entries for this URL.

#### Response Example
```json
{
  "usage_data": [
    {
      "hostname": "example.com",
      "top_llms_txt_urls": [
        {
          "url": "https://example.com/llms.txt",
          "count": 50
        },
        {
          "url": "https://example.com/another_llms.txt",
          "count": 25
        }
      ]
    }
  ]
}
```
```

--------------------------------

### OAuth Configuration and State Management in JavaScript

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/index.html

Defines the OAuth 2.0 configuration constants, application state variables, and references to DOM elements. It also includes functions for initializing the application, handling the OAuth callback, setting the API key, and logging out.

```javascript
const OAUTH_CONFIG = {
  clientId: window.location.hostname,
  redirectUri: window.location.origin + window.location.pathname,
  authorizationEndpoint: 'https://platform.parallel.ai/getKeys/authorize',
  tokenEndpoint: 'https://platform.parallel.ai/getKeys/token',
  scope: 'key:read'
};

let currentApiKey = null;
let extractionData = null;

const marketingView = document.getElementById('marketing-view');
const extractionView = document.getElementById('extraction-view');
const userInfo = document.getElementById('user-info');
const loginBtn = document.getElementById('login-btn');
const logoutBtn = document.getElementById('logout-btn');
const extractionForm = document.getElementById('extraction-form');
const extractBtn = document.getElementById('extract-btn');
const progressSection = document.getElementById('extraction-progress');
const resultsSection = document.getElementById('extraction-results');
const errorSection = document.getElementById('extraction-error');

function init() {
  const urlParams = new URLSearchParams(window.location.search);
  const code = urlParams.get('code');
  const state = urlParams.get('state');

  if (code && state) {
    handleOAuthCallback(code, state);
    return;
  }

  const storedKey = localStorage.getItem('parallel_api_key');
  if (storedKey) {
    setApiKey(storedKey);
    showExtractionView();
  } else {
    showMarketingView();
  }

  loginBtn.addEventListener('click', startOAuthFlow);
  logoutBtn.addEventListener('click', logout);
  extractionForm.addEventListener('submit', handleExtraction);
}

function setApiKey(apiKey) {
  currentApiKey = apiKey;
  localStorage.setItem('parallel_api_key', apiKey);
}

function logout() {
  currentApiKey = null;
  localStorage.removeItem('parallel_api_key');
  showMarketingView();
  resetForm();
}
```

--------------------------------

### Minimum Viable llms.txt Implementation (HTTP Request/Response)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/drafts/SEP-001-discovery.md

This demonstrates the most basic HTTP request and response for serving an llms.txt file from the root path. It ensures backward compatibility and adherence to the primary location requirement.

```http
GET /llms.txt HTTP/1.1
Host: example.com

HTTP/1.1 200 OK
Content-Type: text/markdown
```

--------------------------------

### Download All Files as ZIP (JavaScript)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/extract-from-sitemap/index.html

Bundles all extracted files into a single ZIP archive and initiates a download. It requires the `JSZip` library to be loaded and `extractionData` to be available. The ZIP file is named based on a hostname input field, sanitized for safety.

```javascript
async function downloadAllFilesAsZip() {
  if (!extractionData || !window.JSZip) {
    console.error('JSZip not loaded or no extraction data');
    return;
  }
  try {
    const zip = new JSZip();
    // Add all files to the zip
    Object.entries(extractionData.files).forEach(([path, fileData]) => {
      // Remove leading slash and ensure proper path
      const filename = path.startsWith('/') ? path.substring(1) : path;
      const finalFilename = filename || 'content.md';
      zip.file(finalFilename, fileData.content);
    });
    // Generate zip file
    const content = await zip.generateAsync({ type: 'blob' });
    // Create download link
    const url = URL.createObjectURL(content);
    const a = document.createElement('a');
    a.href = url;
    // Generate filename from hostname
    const hostname = document.getElementById('hostname').value.trim();
    const sanitizedHostname = hostname.replace(/[^a-z0-9.-]/gi, '_');
    a.download = `${sanitizedHostname}_extracted_content.zip`;
    document.body.appendChild(a);
    a.click();
    document.body.removeChild(a);
    URL.revokeObjectURL(url);
  } catch (error) {
    console.error('Error creating ZIP file:', error);
    alert('Error creating ZIP file. Please try again.');
  }
}
```

--------------------------------

### Generate llms.txt from Assets (Bash)

Source: https://context7.com/janwilmake/llmtext-mcp/llms.txt

This bash script provides command-line instructions for generating an llms.txt file from project assets. It outlines the prerequisites, including a manifest file and README.md, and demonstrates how to use npx to run the llms-txt-generate tool, optionally specifying a hostname or output file.

```bash
# Create manifest first (see getassetmanifest section)
node node_modules/getassetmanifest/cli.js

# Generate llms.txt with CNAME file
echo "docs.example.com" > CNAME
npx llms-txt-generate

# Or specify hostname as argument
npx llms-txt-generate --hostname docs.example.com

# Custom output file
npx llms-txt-generate --hostname docs.example.com -o custom.txt

# Input files required:
# - README.md (project documentation)
# - manifest.json (file listing with hashes)
# - CNAME (hostname) OR --hostname argument

# Example output in llms.txt:
# [Content from README.md]
#
# # Files
#
# [getting-started](https://docs.example.com/getting-started.md)
# [api-reference](https://docs.example.com/api-reference.md)
# [configuration](https://docs.example.com/configuration.md)
```

--------------------------------

### URL Validation and UI Handling (JavaScript)

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext/index.html

Handles the user interface interactions for URL validation, including getting input from a text field, triggering the validation process, and updating the UI with loading states and results. It uses `setTimeout` for delayed execution and fetches data from a validation API.

```javascript
function handleCheckUrl() { const input = document.getElementById('llmtext-url'); const url = input.value.trim(); if (!url) { alert('Please enter a URL'); return; } switchTab(1); document.getElementById('check-url').value = url; setTimeout(() => { handleCheckValidation(); }, 100); }
```

--------------------------------

### Server Data Handling and Rendering

Source: https://github.com/janwilmake/llmtext-mcp/blob/main/llmtext/index.html

Manages the display of server data, including filtering, sorting, and rendering in an HTML table. It fetches data from 'index.json', handles search input, and updates the UI based on the data. Dependencies include DOM manipulation and helper functions like formatNumber, getInstallMcpUrl, getFaviconUrl, and renderInvalidDomains.

```javascript
function handleSearch() {
  const searchInput = document.getElementById('search-input');
  const query = searchInput.value.trim();
  if (!window.data || !window.data.servers) return;
  if (query === '') {
    window.filteredServers = null;
    renderServers();
    return;
  }
  const validServers = window.data.servers.filter(server => server.valid);
  window.filteredServers = validServers.filter(server => fuzzyMatch(server.hostname, query));
  renderServers();
}

function renderServers() {
  const tbody = document.getElementById('servers-tbody');
  const stats = document.getElementById('stats');
  const noResults = document.getElementById('no-results');
  const createSuggestion = document.getElementById('create-suggestion');
  const tableContainer = document.querySelector('#tab-2 .table-container');

  if (!window.data || !window.data.servers) {
    stats.innerText = 'LOADING...';
    tbody.innerHTML = '<tr><td colspan="6" style="text-align: center; padding: 2rem;">LOADING DATA...</td></tr>';
    noResults.style.display = 'none';
    createSuggestion.style.display = 'none';
    return;
  }

  const serversToRender = window.filteredServers !== null ? window.filteredServers : window.data.servers.filter(server => server.valid);
  const allValidServers = window.data.servers.filter(server => server.valid);
  stats.innerText = `${formatNumber(allValidServers.length)} MCP SERVERS CREATED`;

  if (serversToRender.length === 0) {
    tbody.innerHTML = '';
    tableContainer.style.display = 'none';
    noResults.style.display = 'block';
    createSuggestion.style.display = 'block';
  } else {
    tableContainer.style.display = 'block';
    noResults.style.display = 'none';
    createSuggestion.style.display = window.filteredServers !== null ? 'block' : 'none';
    tbody.innerHTML = '';
    serversToRender.forEach((server, index) => {
      const tr = document.createElement('tr');
      const mcpUrl = getInstallMcpUrl(server.hostname);
      const llmstxtUrl = `https://${server.hostname}/llms.txt`;
      tr.innerHTML = `
        <td>${index + 1}</td>
        <td>
          <div class="source-name">
            <img src="${getFaviconUrl(server.hostname)}" alt="" class="favicon">
            <a href="${llmstxtUrl}" target="_blank" rel="noopener">${server.hostname}</a>
          </div>
        </td>
        <td>${formatNumber(server.total_tokens)}</td>
        <td>${formatNumber(server.total_requests)}</td>
        <td>${formatNumber(server.unique_users)}</td>
        <td><a href="${mcpUrl}" class="btn-add" target="_blank" rel="noopener">⬇ INSTALL</a></td>
      `;
      tbody.appendChild(tr);
    });
  }
  renderInvalidDomains();
}
```

--------------------------------

### Generate Asset Manifest CLI (Bash)

Source: https://context7.com/janwilmake/llmtext-mcp/llms.txt

Generates an asset manifest from a directory structure using the getassetmanifest command-line interface. This script sets up a sample project structure, creates an `.assetsignore` file for filtering, optionally configures `wrangler.toml`, and then executes the CLI to produce a `manifest.json` file. It adheres to constraints on asset count, total size, and respects ignore patterns.

```bash
# Create project structure
mkdir -p public/docs public/images
echo "# API Docs" > public/docs/api.md
echo "# Guide" > public/docs/guide.md
cp logo.png public/images/

# Create .assetsignore (gitignore format)
cat > public/.assetsignore << 'EOF'
*.tmp
node_modules/
.DS_Store
EOF

# Create wrangler.toml (optional, defines assets directory)
cat > wrangler.toml << 'EOF'
name = "my-docs"

[assets]
directory = "public"
EOF

# Generate manifest
node node_modules/getassetmanifest/cli.js

# Output: manifest.json created in current directory
cat manifest.json

# Expected output:
# {
#   "/docs/api.md": {
#     "hash": "a3b2c1d4e5f6a7b8c9d0e1f2a3b2c1d4",
#     "size": 11
#   },
#   "/docs/guide.md": {
#     "hash": "f6e5d4c3b2a1f0e9d8c7b6a5f6e5d4c3",
#     "size": 8
#   },
#   "/images/logo.png": {
#     "hash": "9f8e7d6c5b4a3f2e1d0c9b8a9f8e7d6c",
#     "size": 15234
#   }
# }

# Constraints:
# - Maximum 20,000 assets
# - Maximum 25 MB total size
# - Respects .assetsignore patterns
# - Auto-ignores: .git, node_modules, .wrangler, .env files
```