### Install Plugin Dependencies

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This command installs the plugin's dependencies using bundle install. This command needs to be run after cloning the plugin and before attempting to run tests or other development tasks. Requires Ruby and Bundler to be installed.

```bash
bundle install
```

--------------------------------

### Manual Installation for Discourse LLMs.txt Plugin in Bash

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Bash commands for manually installing the Discourse LLMs.txt Generator plugin. Includes steps to clone the repository and install the plugin via rake.

```Bash
cd /var/www/discourse/plugins
git clone https://github.com/kaktaknet/discourse-llms-txt-generator.git
cd /var/www/discourse
bundle exec rake plugin:install
```

--------------------------------

### Docker Installation for Discourse LLMs.txt Plugin in Bash

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Bash commands for installing the Discourse LLMs.txt Generator plugin via Docker. Includes steps to add the plugin to app.yml and rebuild the container.

```Bash
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/kaktaknet/discourse-llms-txt-generator.git

# Rebuild container:
cd /var/discourse
./launcher rebuild app
```

--------------------------------

### Example sitemap.xml entries

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/README.md

XML sitemap entries automatically added by the plugin for llms.txt files. Includes priority and change frequency settings for search engine optimization.

```xml
<url>
  <loc>https://your-forum.com/llms.txt</loc>
  <priority>1.0</priority>
  <changefreq>daily</changefreq>
</url>
<url>
  <loc>https://your-forum.com/llms-full.txt</loc>
  <priority>0.9</priority>
  <changefreq>weekly</changefreq>
</url>
```

--------------------------------

### Comment Best Practices and Examples (Ruby)

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Illustrates effective commenting strategies in Ruby code. It highlights when comments are necessary (e.g., for complex logic) and when they are redundant (e.g., for obvious code).

```ruby
# Good - complex regex needs explanation
get "/c/:category_slug_path_with_id/llms.txt" => "discourse_llms_txt/llms#category",
    constraints: { category_slug_path_with_id: /.*/ }

# Bad - obvious code doesn't need comments
def clear_cache
  # This method clears the cache
  Discourse.cache.delete(CACHE_KEY)
end
```

--------------------------------

### Verify and Test Discourse LLMs.txt Plugin in Bash

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Bash commands for verifying the plugin installation and testing endpoints. Includes curl commands to test various LLMs.txt endpoints and monitor logs.

```Bash
curl https://your-forum.com/llms.txt
curl https://your-forum.com/llms-full.txt
curl https://your-forum.com/sitemaps.txt

tail -f /var/www/discourse/logs/production.log | grep llms.txt
```

--------------------------------

### Adding and Running a New Test (Ruby)

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This snippet demonstrates how to add a new test case to the controller specs. It includes setup, action, and assertion phases, and shows how to run a specific test or the entire suite using RSpec.

```bash
spec/requests/discourse_llms_txt/llms_controller_spec.rb
```

```ruby
describe "#index" do
  it "your new test description" do
    # Setup (if needed)
    SiteSetting.some_setting = true

    # Action
    get "/llms.txt"

    # Assertions
    expect(response.status).to eq(200)
    expect(response.body).to include("expected content")
  end
end
```

```bash
bundle exec rspec spec/requests/discourse_llms_txt/llms_controller_spec.rb:LINE_NUMBER
```

```bash
bundle exec rspec plugins/discourse-llms-txt-generator/spec
```

--------------------------------

### Before Hook for Test Setup (Ruby)

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This code block demonstrates the use of a `before` hook in RSpec tests. It's used to set up common state or configurations that are required before each test runs, ensuring a clean and consistent test environment.

```ruby
before do
  SiteSetting.llms_txt_enabled = true
  SiteSetting.llms_txt_allow_indexing = true
end
```

--------------------------------

### Commit Message Format Example

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Provides a standardized format for commit messages to ensure clarity and consistency in version control. It includes a short summary, an optional detailed explanation, and bullet points for additional details.

```markdown
Short summary (50 chars or less)

Detailed explanation if needed. Wrap at 72 characters.

- Bullet points are okay
- Use imperative mood ("Add feature" not "Added feature")
```

--------------------------------

### Enable Verbose Logging in Rails

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Example of how to add verbose logging messages within Rails controller actions and a command to view these logs in real-time.

```ruby
Rails.logger.info("[llms.txt] Debug message: #{variable}")

tail -f log/development.log | grep llms.txt
```

--------------------------------

### Clone Discourse LLM Plugin Repository

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This command clones the discourse-llms-txt-generator plugin repository from GitHub.  Requires Git to be installed and accessible in the system's PATH.  This step retrieves the plugin source code for local development.

```bash
git clone https://github.com/YOUR_USERNAME/discourse-llms-txt-generator.git
```

--------------------------------

### Verify Sitemap Integration for LLM Files

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Commands to fetch the main sitemap and search for entries related to LLM files. This ensures that LLM-generated content is included in the sitemap for search engines.

```bash
# Check main sitemap
curl http://localhost:3000/sitemap.xml

# Search for llms.txt entries
curl http://localhost:3000/sitemap.xml | grep llms
```

--------------------------------

### Verify robots.txt Integration for LLM Files

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Command to fetch and check the robots.txt file for specific 'Allow' or 'Disallow' directives related to LLM documentation files. This helps confirm correct indexing rules.

```bash
# Check if llms.txt entries are in robots.txt
curl http://localhost:3000/robots.txt | grep -A 10 "LLM Documentation"
```

--------------------------------

### Run RSpec Tests with Documentation Format

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This command executes the RSpec tests and outputs the results in a verbose documentation format, providing detailed information about each test case and its outcome.  Useful for detailed debugging and reporting.

```bash
bundle exec rspec plugins/discourse-llms-txt-generator/spec/requests/discourse_llms_txt/llms_controller_spec.rb --format documentation
```

--------------------------------

### Manual Testing of Generated Files (Bash)

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This snippet shows the bash commands used to manually test the different generated TXT files by making HTTP requests to the Discourse server. It covers navigation, full content, sitemaps, and category/topic/tag-specific files.

```bash
# Start Discourse development server
cd /var/www/discourse
bundle exec rails s

# In another terminal, test endpoints
curl http://localhost:3000/llms.txt
curl http://localhost:3000/llms-full.txt
curl http://localhost:3000/sitemaps.txt
curl http://localhost:3000/c/general/1/llms.txt
curl http://localhost:3000/t/welcome-topic/1/llms.txt
curl http://localhost:3000/tag/announcement/llms.txt
```

--------------------------------

### Configuration Settings

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Allows configuration of plugin behavior through the Discourse admin panel.

```APIDOC
## Configuration Settings

Configure plugin behavior through Discourse admin panel.

### Settings

- **llms_txt_enabled** (boolean)
  - **Description:** Enable or disable the plugin.
  - **Default:** `true`

- **llms_txt_allow_indexing** (boolean)
  - **Description:** Allow AI crawler indexing for llms.txt files.
  - **Default:** `true`

- **llms_txt_blocked_user_agents** (string)
  - **Description:** A comma-separated list of user agents to block from accessing llms.txt files only.
  - **Example:** `"Omgilibot, ChatGPT-User, CCBot"`

- **llms_txt_intro_text** (string)
  - **Description:** Custom introductory text to be displayed at the beginning of llms.txt files.
  - **Example:** `"This forum provides discussions, support, and knowledge sharing."`

- **llms_txt_full_description** (string)
  - **Description:** Optional custom description for llms-full.txt, displayed at the top.
  - **Example:** `"A technical support community for XYZ Software users..."`
```

--------------------------------

### Run RSpec Tests - Full Suite

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This command executes the entire RSpec test suite for the discourse-llms-txt-generator plugin. It provides a comprehensive check of the plugin's functionality and ensures all tests are passing.

```bash
bundle exec rspec plugins/discourse-llms-txt-generator/spec
```

--------------------------------

### Check Enabled Plugins in Rails Console

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Ruby code to list the names of all currently enabled Discourse plugins, useful for verifying if the LLM TXT plugin is loaded.

```ruby
rails c
> Discourse.plugins.map(&:name)
```

--------------------------------

### Fabricate Test Data

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

These commands use Discourse's Fabricate framework to create test data. The `fab!` method creates test data once per test suite to optimize performance rather than before each test.

```ruby
fab!(:category) { Fabricate(:category) }
```

```ruby
fab!(:topic) { Fabricate(:topic, views: 100) }
```

```ruby
fab!(:post) { Fabricate(:post, topic: topic) }
```

```ruby
fab!(:tag) { Fabricate(:tag) }
```

--------------------------------

### Refresh Sitemap and Robots.txt with Rake Tasks

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Commands to refresh the sitemap and clear the robots.txt cache using Discourse's rake tasks. Recommended for ensuring the latest content is reflected in search engine indexing.

```bash
cd /var/www/discourse

# Option 1: Use the plugin's rake task (recommended)
bundle exec rake llms_txt:refresh

# Option 2: Manually regenerate sitemap and clear robots cache
bundle exec rake sitemap:refresh
rails runner "Rails.cache.delete('robots_txt')"

# Option 3: Restart Discourse (clears all caches)
cd /var/discourse
./launcher restart app
```

--------------------------------

### Reset Test Database for Rails

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Bash command to drop, create, and migrate the test database, which can resolve common issues like database errors during test runs.

```bash
RAILS_ENV=test bundle exec rake db:drop db:create db:migrate
```

--------------------------------

### Test LLMs TXT File Generation with Curl

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/README.md

Uses curl commands to manually test the generation and accessibility of llms.txt and llms-full.txt files. Includes testing with disabled settings to verify 404 responses.

```bash
# Test navigation file
curl https://your-forum.com/llms.txt

# Test full content file
curl https://your-forum.com/llms-full.txt

# Test with disabled setting
# Admin → Settings → llms_txt_enabled = false
curl https://your-forum.com/llms.txt
# Should return 404
```

--------------------------------

### Testing Settings Enforcement

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This code snippet demonstrates how to test the enforcement of plugin settings, verifying that the plugin's behavior changes based on setting values using RSpec.

```ruby
SiteSetting.llms_txt_enabled = false
```

```ruby
get "/llms.txt"
```

```ruby
expect(response.status).to eq(404)
```

--------------------------------

### Check Plugin Settings with Rails Console

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Ruby code snippets to execute within the Rails console to check the enabled status and indexing permissions of the LLM TXT plugin.

```ruby
rails c
> SiteSetting.llms_txt_enabled
> SiteSetting.llms_txt_allow_indexing
```

--------------------------------

### Testing Suite Overview in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Indicates the presence of a comprehensive RSpec testing suite for the plugin. This implies thorough test coverage for all functionalities, ensuring reliability and stability.

```ruby
# Comprehensive RSpec test coverage for all plugin functionality.
```

--------------------------------

### Check Plugin Status with Rake Task

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

A command to quickly check the status of the LLM TXT plugin, including its enabled status, indexing permissions, and test endpoint URLs.

```bash
# Quick check of plugin configuration
bundle exec rake llms_txt:check
```

--------------------------------

### Testing Access Tracking

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This code snippet demonstrates how to test access tracking within the plugin, verifying that the access count is incremented correctly using RSpec.

```ruby
expect { get "/llms.txt" }.to change { PluginStore.get(DiscourseLlmsTxt::PLUGIN_NAME, "access_count_index").to_i }.by(1)
```

--------------------------------

### Run Specific RSpec Test File

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This command runs a specific test file using the RSpec framework. It's useful for focused testing of particular features or components of the plugin.

```bash
bundle exec rspec plugins/discourse-llms-txt-generator/spec/requests/discourse_llms_txt/llms_controller_spec.rb
```

--------------------------------

### Analytics Tracking with PluginStore in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Implements analytics tracking for access to LLM text files using the Discourse PluginStore. It records access counts and last access timestamps for various file types, providing insights into usage patterns.

```ruby
# Access tracking in controller:
def track_access
  key = "access_count_#{action_name}"
  current_count = PluginStore.get(DiscourseLlmsTxt::PLUGIN_NAME, key).to_i
  PluginStore.set(DiscourseLlmsTxt::PLUGIN_NAME, key, current_count + 1)

  PluginStore.set(
    DiscourseLlmsTxt::PLUGIN_NAME,
    "last_access_#{action_name}",
    Time.now.to_i
  )
end

# Retrieve analytics via Rails console:
# Access count for navigation file
PluginStore.get("discourse-llms-txt-generator", "access_count_index")
# => 1523

# Access count for full content file
PluginStore.get("discourse-llms-txt-generator", "access_count_full")
# => 892

# Access count for sitemaps
PluginStore.get("discourse-llms-txt-generator", "access_count_sitemaps")
# => 45

# Last access timestamp
timestamp = PluginStore.get("discourse-llms-txt-generator", "last_access_index")
Time.at(timestamp)
# => 2025-11-09 14:23:45 UTC

# Access count for category files
PluginStore.get("discourse-llms-txt-generator", "access_count_category")

# Access count for topic files
PluginStore.get("discourse-llms-txt-generator", "access_count_topic")

# Access count for tag files
PluginStore.get("discourse-llms-txt-generator", "access_count_tag")

# User agent tracking via server logs:
# tail -f /var/www/discourse/logs/production.log | grep "llms.txt"
```

--------------------------------

### Generate Discourse Main Navigation llms.txt

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Generates the primary llms.txt navigation file for the Discourse forum. This file includes site metadata, a hierarchy of categories, and a list of the latest topics. It is cached for performance.

```ruby
# Generate cached navigation file
content = DiscourseLlmsTxt::Generator.generate_navigation

# Example output structure:
# # Forum Title
# > Site description
#
# Introduction text...
#
# ## Categories and Subcategories
# ### [General Discussion](url)
# Description of category
#
# - [Help & Support](url): Support subcategory
#
# ## Latest Topics
# - [Topic Title](url) - Category Name (2025-11-09)
#
# ## Additional Resources
# - [Full Documentation (llms-full.txt)](url): Complete forum content
```

--------------------------------

### Sitemaps Index Generation

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Generates a complete index of all available llms.txt resources across the forum, respecting privacy settings and listing them in robots.txt.

```APIDOC
## Sitemaps Index Generation

Generates complete index of all available llms.txt resources across forum.

### Method
GET

### Endpoint
/sitemaps.txt

### Description
Retrieves a sitemap index of all llms.txt resources.

### Response
#### Success Response (200)
- **content** (string) - A list of URLs to llms.txt resources.

### Response Example
```
https://forum.example.com/llms.txt
https://forum.example.com/llms-full.txt
https://forum.example.com/c/general/1/llms.txt
https://forum.example.com/tag/tutorial/llms.txt
```
```

--------------------------------

### Discourse LLMs TXT HTTP Controller Endpoints (Ruby)

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Handles HTTP requests for `llms.txt` files within a Discourse forum, including authentication, permission checks, and analytics tracking. It defines routes for various `llms.txt` endpoints like index, full, category, topic, and tag specific files.

```ruby
# Routes defined in plugin.rb:
# GET /llms.txt                              => index action
# GET /llms-full.txt                         => full action
# GET /sitemaps.txt                          => sitemaps action
# GET /c/:category_slug_path_with_id/llms.txt => category action
# GET /t/:topic_slug/:topic_id/llms.txt      => topic action
# GET /tag/:tag_name/llms.txt                => tag action

# Example usage in controller:
class LlmsController < ApplicationController
  def index
    content = DiscourseLlmsTxt::Generator.generate_navigation
    render plain: content, content_type: "text/plain; charset=utf-8"
  end

  def category
    category = Category.find_by_slug_path_with_id(params[:category_slug_path_with_id])
    return render_404 unless category && guardian.can_see?(category)

    canonical_url = "#{Discourse.base_url}/c/#{CGI.escape(category.slug)}/#{category.id}"
    response.headers['Link'] = "<#{canonical_url}>; rel=\"canonical\""

    content = DiscourseLlmsTxt::Generator.generate_category_llms(category)
    render plain: content, content_type: "text/plain; charset=utf-8"
  end
end

# Security features:
# - Guardian permission checks for private content
# - CSRF protection skipped (public endpoints like robots.txt)
# - Returns 404 for disabled plugin or unauthorized access
# - Returns 403 when indexing is disabled
# - Tracks access analytics via PluginStore
```

--------------------------------

### HTTP Controller Endpoints

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Handles all HTTP requests for llms.txt files, including authentication, permission checks, and analytics tracking.

```APIDOC
## HTTP Controller Endpoints

Handles all HTTP requests for llms.txt files with authentication, permission checks, and analytics tracking.

### Endpoints

- **GET /llms.txt**
  - **Description:** Retrieves the main llms.txt navigation.
  - **Response Example:**
    ```
    # llms.txt content
    ```

- **GET /llms-full.txt**
  - **Description:** Retrieves the full llms.txt content.
  - **Response Example:**
    ```
    # llms-full.txt content
    ```

- **GET /c/:category_slug_path_with_id/llms.txt**
  - **Description:** Retrieves llms.txt content for a specific category.
  - **Path Parameters:**
    - **category_slug_path_with_id** (string) - Required - The slug path and ID of the category.
  - **Response Example:**
    ```
    # llms.txt content for the category
    ```

- **GET /t/:topic_slug/:topic_id/llms.txt**
  - **Description:** Retrieves llms.txt content for a specific topic.
  - **Path Parameters:**
    - **topic_slug** (string) - Required - The slug of the topic.
    - **topic_id** (integer) - Required - The ID of the topic.
  - **Response Example:**
    ```
    # llms.txt content for the topic
    ```

- **GET /tag/:tag_name/llms.txt**
  - **Description:** Retrieves llms.txt content for a specific tag.
  - **Path Parameters:**
    - **tag_name** (string) - Required - The name of the tag.
  - **Response Example:**
    ```
    # llms.txt content for the tag
    ```

### Security & Error Handling
- Returns 404 if the plugin is disabled or access is unauthorized.
- Returns 403 if indexing is disabled.
- Respects user permissions and privacy settings.
```

--------------------------------

### Generate Discourse Full Content Index llms.txt

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Generates a comprehensive forum index file (llms-full.txt) containing all public topics. This file can optionally include excerpts from posts and detailed category descriptions. It is not cached due to its potentially large size.

```ruby
# Generate full content file (not cached due to size)
content = DiscourseLlmsTxt::Generator.generate_full_content

# Example with excerpts disabled (default):
# # Forum Title - Full Content
# > Site description
#
# [← Back to Navigation (llms.txt)](url)
#
# ## Topics
# **[Category Name](url)** - [Topic Title](url)
# **[Another Category](url)** - [Another Topic](url)

# Example with excerpts enabled:
# **[Category Name](url)** - [Topic Title](url)
#   > This is the first 500 characters of the post content...
#
# **[Another Category](url)** - [Another Topic](url)
#   > Another post excerpt here...
```

--------------------------------

### Testing HTTP Responses

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

These code snippets demonstrate how to test HTTP responses, asserting the status code and content type using RSpec.  This verifies the plugin returns expected responses under certain conditions.

```ruby
get "/llms.txt"
```

```ruby
expect(response.status).to eq(200)
```

```ruby
expect(response.content_type).to include("text/plain")
```

--------------------------------

### Configure blocked user agents

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/README.md

Plugin setting for specifying which user agents should be blocked from accessing llms.txt files. Multiple bot names can be specified as comma-separated values.

```text
Omgilibot, ChatGPT-User, AnotherBot
```

--------------------------------

### Test LLMs.txt Navigation File Generation in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

RSpec tests for the LLMs.txt navigation file generation. Verifies response status, content type, and body content including site title, categories, and latest topics. Also tests plugin disablement and access count tracking.

```Ruby
RSpec.describe DiscourseLlmsTxt::LlmsController do
  describe "#index" do
    it "returns llms.txt navigation file" do
      get "/llms.txt"

      expect(response.status).to eq(200)
      expect(response.content_type).to include("text/plain")
      expect(response.body).to include(SiteSetting.title)
      expect(response.body).to include("## Categories and Subcategories")
      expect(response.body).to include("## Latest Topics")
    end

    it "returns 404 when plugin is disabled" do
      SiteSetting.llms_txt_enabled = false
      get "/llms.txt"
      expect(response.status).to eq(404)
    end

    it "tracks access count" do
      expect {
        get "/llms.txt"
      }.to change {
        PluginStore.get(DiscourseLlmsTxt::PLUGIN_NAME, "access_count_index").to_i
      }.by(1)
    end
  end
```

--------------------------------

### Verify Robots.txt Hook with Rails Console

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Ruby code to trigger the robots.txt event in the Rails console, useful for verifying that the plugin's hook for robots.txt is functioning correctly.

```ruby
rails c
> DiscourseEvent.trigger(:robots_txt, "")
```

--------------------------------

### Content Filtering and Access Settings in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Demonstrates how to configure content filtering settings for LLM text generation and programmatically access these settings. It includes logic for determining post limits based on a string value and conditional content generation.

```ruby
# Content filtering settings
llms_txt_min_views: 50                # Minimum topic views for inclusion
llms_txt_posts_limit: "medium"        # small=500, medium=2500, large=5000, all=unlimitedllms_txt_include_excerpts: false      # Include post excerpts (increases file size)llms_txt_post_excerpt_length: 500     # Maximum excerpt length (100-5000 chars)llms_txt_latest_topics_count: 50      # Latest topics in navigation file (max 100)

# Cache settings
llms_txt_cache_minutes: 60            # Cache duration for navigation file

# Accessing settings programmatically:
if SiteSetting.llms_txt_enabled && SiteSetting.llms_txt_allow_indexing
  content = DiscourseLlmsTxt::Generator.generate_navigation
end

# Get posts limit value:
limit = case SiteSetting.llms_txt_posts_limit
when "small" then 500
when "medium" then 2500
when "large" then 5000
when "all" then nil
end
```

--------------------------------

### Discourse LLMs TXT Configuration Settings (YAML)

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Defines configuration settings for the Discourse LLMs TXT plugin, managed through the Discourse admin panel. These settings control the plugin's behavior, including enabling/disabling the plugin, allowing indexing, blocking specific user agents, and setting custom text for the generated files.

```yaml
# Settings defined in config/settings.yml

# Enable/disable plugin
llms_txt_enabled: true

# Allow AI crawler indexing
llms_txt_allow_indexing: true

# Block specific bots from llms.txt files only
llms_txt_blocked_user_agents: "Omgilibot, ChatGPT-User, CCBot"
# These bots can still access main forum, just not llms.txt files

# Custom introduction text for llms.txt
llms_txt_intro_text: "This forum provides discussions, support, and knowledge sharing."

# Optional custom description for llms-full.txt (appears at top)
llms_txt_full_description: "A technical support community for XYZ Software users..."
```

--------------------------------

### Robots.txt Integration and Bot Blocking in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Configures robots.txt integration to manage search engine and bot access to LLM documentation files. It allows administrators to specify which bots are allowed or disallowed from crawling specific files based on various settings.

```ruby
# robots.txt entries automatically added when enabled:

# When llms_txt_allow_indexing = true:
# LLM Documentation Files
Allow: /llms.txt
Allow: /llms-full.txt
Sitemap: https://forum.example.com/sitemaps.txt

# When llms_txt_blocked_user_agents is set:
User-agent: Omgilibot
Disallow: /llms.txt
Disallow: /llms-full.txt

User-agent: ChatGPT-User
Disallow: /llms.txt
Disallow: /llms-full.txt

User-agent: CCBot
Disallow: /llms.txt
Disallow: /llms-full.txt

# When llms_txt_allow_indexing = false:
Disallow: /llms.txt
Disallow: /llms-full.txt

# Configure bot blocking:
# Admin → Settings → Plugins → discourse-llms-txt-generator
# llms_txt_blocked_user_agents: "Omgilibot, ChatGPT-User, CCBot"

# Common bots to consider blocking:
# - Omgilibot: Omgili web crawler
# - ChatGPT-User: OpenAI's ChatGPT crawler
# - CCBot: Common Crawl bot
# - anthropic-ai: Anthropic's crawler
# - Google-Extended: Google's AI training crawler
```

--------------------------------

### Generate Dynamic Topic llms.txt

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Generates a complete llms.txt file for a specific topic, including all its posts in markdown format. This is useful for AI systems that need detailed content from individual discussions. The content is dynamically generated and served via HTTP requests.

```ruby
# Find topic and generate its llms.txt
topic = Topic.find_by(id: 456)
content = DiscourseLlmsTxt::Generator.generate_topic_llms(topic)

# HTTP request example:
# GET https://forum.example.com/t/installation-guide/456/llms.txt
#
# Returns:
# # Complete Installation Guide
#
# **Category:** [Support](url)
# **Created:** 2025-11-09 10:30 UTC
# **Views:** 1523
# **Replies:** 12
# **URL:** https://forum.example.com/t/installation-guide/456
#
# ---
#
# ## Post #1 by @admin
#
# This guide will walk you through installing the software.
#
# ### Prerequisites
#
# - Ubuntu 20.04 or higher
# - Root access
#
# [Full markdown content of post]
#
# ---
#
# ## Post #2 by @user123
#
# Thanks for this guide! I had an issue with permissions...
#
# [Full markdown content]
#
# ---
#
# **Canonical:** https://forum.example.com/t/installation-guide/456
# **Original content:** https://forum.example.com/t/installation-guide/456
```

--------------------------------

### Cache Management in Rails Console (Ruby)

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Demonstrates how to interact with the cache for the discourse-llms-txt-generator plugin using the Rails console. It includes commands to clear the cache, read cache status, and check the last update timestamp.

```ruby
# Rails console
rails c

# Clear cache
DiscourseLlmsTxt::Generator.clear_cache

# Check cache status
Discourse.cache.read("llms_txt_navigation")

# Check last update timestamp
PluginStore.get(DiscourseLlmsTxt::PLUGIN_NAME, "last_cache_update")
```

--------------------------------

### Access Analytics Data in Rails Console

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/README.md

Retrieves access statistics for llms.txt files using PluginStore. Shows access counts and last access timestamps for both index and full content files.

```ruby
PluginStore.get("discourse-llms-txt-generator", "access_count_index")
PluginStore.get("discourse-llms-txt-generator", "access_count_full")
PluginStore.get("discourse-llms-txt-generator", "last_access_index")
```

--------------------------------

### Smart Cache Management for LLMs TXT (Ruby)

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Implements an intelligent caching system for `llms.txt` content that only regenerates when new activity occurs on the forum. This includes a scheduled job to check for updates, manual cache clearing functions, and real-time cache invalidation upon content changes (post creation/editing).

```ruby
# Scheduled job runs every hour
class UpdateLlmsTxtCache < Jobs::Scheduled
  every 1.hour

  def execute(args)
    return unless SiteSetting.llms_txt_enabled

    if DiscourseLlmsTxt::Generator.should_update_cache?
      Rails.logger.info("[llms.txt] Updating cache due to new content")

      DiscourseLlmsTxt::Generator.clear_cache
      DiscourseLlmsTxt::Generator.generate_navigation
      DiscourseLlmsTxt::Generator.update_cache_timestamp

      Rails.logger.info("[llms.txt] Cache updated successfully")
    else
      Rails.logger.debug("[llms.txt] No new content, skipping cache update")
    end
  end
end

# Check if cache update is needed:
def should_update_cache?
  last_check = Rails.cache.read(CACHE_KEY_LAST_CHECK)
  return true if last_check.nil? || last_check < 1.hour.ago

  last_topic = Topic.maximum(:created_at)
  last_category = Category.maximum(:updated_at)

  return true if last_topic && last_topic > last_check
  return true if last_category && last_category > last_check

  false
end

# Manual cache clearing:
DiscourseLlmsTxt::Generator.clear_cache

# Real-time cache invalidation on content changes:
on(:post_created) do |post|
  DiscourseLlmsTxt::Generator.clear_cache if SiteSetting.llms_txt_enabled
end

on(:post_edited) do |post|
  DiscourseLlmsTxt::Generator.clear_cache if SiteSetting.llms_txt_enabled
end
```

--------------------------------

### Run Specific RSpec Test by Line Number

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

This command executes a specific test case within an RSpec file by specifying the line number. This is helpful for pinpointing and testing individual test cases in detail.

```bash
bundle exec rspec plugins/discourse-llms-txt-generator/spec/requests/discourse_llms_txt/llms_controller_spec.rb:53
```

--------------------------------

### Troubleshoot Robots.txt Cache with Rails Runner

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Command to clear the robots.txt cache, which can help resolve issues where changes to robots.txt rules are not being reflected.

```bash
# Clear robots.txt cache:
rails runner "Rails.cache.delete('robots_txt')"
```

--------------------------------

### Troubleshoot Sitemap Issues with Rails Runner and Rake

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Commands to clear the sitemap cache and force sitemap regeneration, which can resolve issues where LLM TXT files are not appearing in the sitemap.

```bash
# Clear sitemap cache:
rails runner "SitemapUrl.delete_all"
bundle exec rake sitemap:refresh

# Force sitemap regeneration:
bundle exec rake sitemap:regenerate
```

--------------------------------

### Generate Dynamic Category llms.txt

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Generates an on-demand llms.txt file for a specific category. This file includes subcategories and all topics within that category. The content is dynamically generated and served via HTTP requests, with canonical headers for SEO.

```ruby
# Find category and generate its llms.txt
category = Category.find_by_slug_path_with_id("general-discussion/1")
content = DiscourseLlmsTxt::Generator.generate_category_llms(category)

# HTTP request example:
# GET https://forum.example.com/c/support/2/llms.txt
#
# Returns:
# # Support
# > Category: Forum Title
#
# Get help with installation and configuration.
#
# **Category URL:** https://forum.example.com/c/support/2
#
# ## Subcategories
#
# - [Installation Help](url): Installation guides
# - [Configuration](url): Configuration questions
#
# ## Topics
#
# - [How to install on Ubuntu](url) (1523 views, 45 replies)
# - [Common errors](url) (892 views, 23 replies)
#
# **Canonical:** https://forum.example.com/c/support/2
# **Original content:** https://forum.example.com/c/support/2

# Controller responds with canonical header:
# Link: <https://forum.example.com/c/support/2>; rel="canonical"
```

--------------------------------

### Generate Dynamic Tag llms.txt

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Generates an llms.txt file for all topics associated with a specific tag. This allows AI systems to discover content related to a particular subject across the forum. The content is dynamically generated and served via HTTP requests.

```ruby
# Find tag and generate its llms.txt
tag = Tag.find_by_name("tutorial")
content = DiscourseLlmsTxt::Generator.generate_tag_llms(tag)

# HTTP request example:
# GET https://forum.example.com/tag/tutorial/llms.txt
#
# Returns:
# # Tag: tutorial
# > Forum Title
#
# **Tag URL:** https://forum.example.com/tag/tutorial
```

--------------------------------

### Generate LLMs Sitemap Index (Ruby)

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Generates a complete index of all available `llms.txt` resources across the forum. This process is cached for performance and respects privacy settings, only including public resources. The generated index is typically listed in `robots.txt`.

```ruby
# Generate sitemap index (cached)
content = DiscourseLlmsTxt::Generator.generate_sitemaps

# Example output:
# https://forum.example.com/llms.txt
# https://forum.example.com/llms-full.txt
# https://forum.example.com/c/general/1/llms.txt
# https://forum.example.com/c/support/2/llms.txt
# https://forum.example.com/c/support/installation/10/llms.txt
# https://forum.example.com/t/welcome-post/123/llms.txt
# https://forum.example.com/t/installation-guide/456/llms.txt
# https://forum.example.com/tag/announcement/llms.txt
# https://forum.example.com/tag/tutorial/llms.txt

# Respects privacy settings - only public resources included
# Listed in robots.txt as: Sitemap: https://forum.example.com/sitemaps.txt
```

--------------------------------

### Smart Cache Management

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Manages an intelligent caching system that regenerates content only when the forum has new activity.

```APIDOC
## Smart Cache Management

Intelligent caching system that only regenerates content when forum has new activity.

### Cache Update

- **Mechanism:** Scheduled job runs every hour.
- **Condition:** Updates cache if `llms_txt_enabled` is true and new content (topics or categories) has been created or updated since the last check.
- **Actions:** Clears existing cache, regenerates navigation, and updates the cache timestamp.

### Cache Invalidation

- **Manual:** `DiscourseLlmsTxt::Generator.clear_cache`
- **Automatic (Real-time):** Cache is cleared on `post_created` and `post_edited` events if `llms_txt_enabled` is true.

### Cache Check Logic

```ruby
def should_update_cache?
  last_check = Rails.cache.read(CACHE_KEY_LAST_CHECK)
  return true if last_check.nil? || last_check < 1.hour.ago

  last_topic = Topic.maximum(:created_at)
  last_category = Category.maximum(:updated_at)

  return true if last_topic && last_topic > last_check
  return true if last_category && last_category > last_check

  false
end
```
```

--------------------------------

### Block LLM bots in robots.txt

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/README.md

Configuration to block specific user agents from accessing llms.txt files. This prevents certain LLM crawlers from indexing content while allowing normal forum crawling.

```robots.txt
User-agent: Omgilibot
Disallow: /llms.txt
Disallow: /llms-full.txt

User-agent: ChatGPT-User
Disallow: /llms.txt
Disallow: /llms-full.txt
```

--------------------------------

### Monitor cache updates via logs

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/README.md

Command to monitor llms.txt cache update activity in Discourse production logs. Shows when cache is updated due to new content or skipped when no changes exist.

```bash
tail -f /var/www/discourse/logs/production.log | grep llms.txt
```

--------------------------------

### Testing Content Inclusion

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

These code snippets show how to test content inclusion in responses using RSpec. It asserts that expected text is included in the response body and unexpected text is not.

```ruby
expect(response.body).to include("Expected text")
```

```ruby
expect(response.body).not_to include("Unexpected text")
```

--------------------------------

### Test Category LLMs.txt with Canonical URLs in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

RSpec tests for category-specific LLMs.txt files. Verifies the inclusion of canonical URLs in both the Link header and response body for SEO compliance.

```Ruby
describe "#category" do
    fab!(:category) { Fabricate(:category) }

    it "includes canonical URL in Link header" do
      get "/c/#{category.slug}/#{category.id}/llms.txt"

      canonical_url = "#{Discourse.base_url}/c/#{category.slug}/#{category.id}"
      expect(response.headers['Link']).to eq("<#{canonical_url}>; rel=\"canonical\"")
    end

    it "includes canonical URL in response body" do
      get "/c/#{category.slug}/#{category.id}/llms.txt"

      canonical_url = "#{Discourse.base_url}/c/#{category.slug}/#{category.id}"
      expect(response.body).to include("**Canonical:** #{canonical_url}")
    end
  end
```

--------------------------------

### Test Topic LLMs.txt Generation in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

RSpec tests for topic-specific LLMs.txt files. Verifies the response status and the inclusion of topic title and post content in the response body.

```Ruby
describe "#topic" do
    fab!(:topic) { Fabricate(:topic) }
    fab!(:post) { Fabricate(:post, topic: topic) }

    it "returns topic llms.txt" do
      get "/t/#{topic.slug}/#{topic.id}/llms.txt"

      expect(response.status).to eq(200)
      expect(response.body).to include(topic.title)
      expect(response.body).to include(post.raw)
    end
  end
```

--------------------------------

### Clear and Check Cache for Discourse LLMs.txt Plugin in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Ruby commands for clearing and checking the cache status of the Discourse LLMs.txt Generator plugin. Includes commands to clear cache and check last update time.

```Ruby
DiscourseLlmsTxt::Generator.clear_cache

last_update = DiscourseLlmsTxt::Generator.last_update_time
# => 2025-11-09 14:30:00 UTC
```

--------------------------------

### Private Category and Topic Filtering in Ruby

Source: https://context7.com/kaktaknet/discourse-llms-txt-generator/llms.txt

Implements privacy and security features by filtering out private categories and topics at the SQL level and through Guardian permission checks. It ensures that restricted content is not exposed in public LLM files and returns 404 errors for unauthorized access.

```ruby
# Private category filtering (SQL-level):
topics = Topic.visible
  .where(archetype: "regular")
  .joins(:category)
  .where(categories: { read_restricted: false })  # Only public categories
  .where("topics.views >= ?", SiteSetting.llms_txt_min_views)

# Guardian permission checks for dynamic files:
def category
  category = Category.find_by_slug_path_with_id(params[:category_slug_path_with_id])
  return render_404 unless category && guardian.can_see?(category)
  # Returns 404 if user can't access category
end

def topic
  topic = Topic.find_by(id: params[:topic_id])
  return render_404 unless topic && guardian.can_see?(topic)
  # Returns 404 if topic is private or deleted
end

# Example: Private category protection
private_category = Category.create!(
  name: "Staff Only",
  read_restricted: true,
  user: admin
)

# This category and its topics will NEVER appear in:
# - /llms.txt
# - /llms-full.txt
# - /sitemaps.txt
# - /tag/*/llms.txt

# Direct access returns 404:
# GET /c/staff-only/123/llms.txt => 404 Not Found
```

--------------------------------

### Manually Clear Plugin Cache in Rails

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/CONTRIBUTING.md

Ruby code to manually clear the cache used by the LLM TXT Generator plugin, useful for ensuring settings or generated content updates are applied.

```ruby
rails c
> DiscourseLlmsTxt::Generator.clear_cache
```

--------------------------------

### Clear LLMs TXT Cache in Rails Console

Source: https://github.com/kaktaknet/discourse-llms-txt-generator/blob/main/README.md

Manually clears the cache for LLMs TXT file generation through the Rails console. Useful for development and troubleshooting caching issues.

```ruby
# In Rails console
DiscourseLlmsTxt::Generator.clear_cache
```