### LlmsSection Usage Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/03-llms-api.md Demonstrates how to structure the 'sections' field within LlmsOptions to create a 'Getting Started' section with installation and tutorial links. ```typescript { sections: [ { title: 'Getting Started', links: [ { title: 'Installation', url: '/install' }, { title: 'Tutorial', url: '/tutorial' } ] } ] } ``` -------------------------------- ### robots.txt Output Example Source: https://github.com/casoon/astro-site-files/blob/main/README.md This is an example of the generated robots.txt content based on the configuration options. ```text User-agent: * Disallow: /admin Disallow: /private/ Allow: /admin/public/ Crawl-delay: 2 User-agent: Googlebot Crawl-delay: 1 Sitemap: https://example.com/sitemap.xml ``` -------------------------------- ### RSS Module Import Examples Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Provides examples for importing RSS-related functions and types for both build-time rendering and API routes. ```typescript // Build-time RSS (via sitemap.rss) import { renderRssFeed } from '@casoon/astro-site-files' // API route RSS import { createRssRoute } from '@casoon/astro-site-files/rss' import type { CreateRssRouteOptions } from '@casoon/astro-site-files/rss' ``` -------------------------------- ### Install @casoon/astro-site-files Source: https://github.com/casoon/astro-site-files/blob/main/README.md Install the package using npm. Ensure your Node.js and Astro versions meet the requirements. ```sh npm install @casoon/astro-site-files ``` -------------------------------- ### Example Agent Rules Configuration Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/02-robots-api.md Demonstrates how to configure specific rules for different bots using the AgentRule interface. This example sets crawl delays and disallows paths for Googlebot, and sets crawl delays for Bingbot and DuckDuckBot. ```typescript renderRobotsTxt({ agents: [ { userAgent: 'Googlebot', crawlDelay: 1, disallow: ['/admin/'] }, { userAgent: ['Bingbot', 'DuckDuckBot'], crawlDelay: 2 } ] }, 'https://example.com') ``` -------------------------------- ### Astro Integration Setup Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Demonstrates how to set up the `@casoon/astro-site-files` integration in an Astro project's configuration file. ```typescript // astro.config.ts import { defineConfig } from 'astro/config' import siteFiles from '@casoon/astro-site-files' export default defineConfig({ site: 'https://example.com', integrations: [siteFiles({ /* options */ })] }) ``` -------------------------------- ### I18n Configuration Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/04-sitemap-api.md Example of how to configure i18n settings for multi-language sites, including the default locale and a mapping of locale prefixes to language codes. ```javascript { i18n: { defaultLocale: 'en', locales: { 'en': 'en-US', 'de': 'de-DE', 'fr': 'fr-FR' } } } ``` -------------------------------- ### Priority Rule Usage Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/04-sitemap-api.md Example of how to configure priority rules in the sitemap configuration. Assigns priorities to different URL patterns. ```typescript { priority: [ { pattern: '/blog/', priority: 0.9 }, { pattern: '/important/', priority: 0.95 }, { pattern: /^\/archive\//, priority: 0.5 } ] } ``` -------------------------------- ### Example Robots.txt Output Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/02-robots-api.md Illustrates the structure of the generated robots.txt file, including user-agent blocks, per-bot rules, and the sitemap directive. ```text User-agent: * Disallow: /admin Disallow: /private/ Allow: /admin/public/ Crawl-delay: 2 User-agent: GPTBot Disallow: / User-agent: Googlebot Crawl-delay: 1 Sitemap: https://example.com/sitemap.xml ``` -------------------------------- ### RssItem Usage Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/05-rss-api.md Demonstrates how to create an array of RssItem objects for use in an RSS feed. Shows examples with required and optional fields, including custom data. ```typescript const items: RssItem[] = [ { title: 'First Post', description: 'This is my first post about ', pubDate: new Date('2026-06-24'), link: '/blog/first/', author: 'Alice', categories: ['tutorial', 'astro'], }, { title: 'Second Post', description: 'Follow-up discussion', pubDate: '2026-06-25', link: 'https://example.com/blog/second/', customData: '' } ] ``` -------------------------------- ### Sitemap Source Usage Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/04-sitemap-api.md Example of defining custom sitemap sources using async functions. Fetches data from collections to generate dynamic sitemap entries. ```typescript { sources: [ async () => { const posts = await getCollection('blog') return posts.map(p => ({ loc: `/blog/${p.id}/`, lastmod: p.data.date.toISOString().split('T')[0] })) }, async () => { const pages = await getCollection('pages') return pages.map(p => ({ loc: `/pages/${p.id}/`, priority: 0.8 })) } ] } ``` -------------------------------- ### Usage Example: Dynamic Content Generation Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/03-llms-api.md Demonstrates how to configure `LlmsOptions` with multiple dynamic sources, fetching data from Astro collections for documentation and blog posts. ```typescript import { getCollection } from 'astro:content' const llmsOpts: LlmsOptions = { title: 'My Site', description: 'Documentation and blog.', sources: [ async () => { const docs = await getCollection('docs') return { title: 'Documentation', links: docs.map(doc => ({ title: doc.data.title, url: `/docs/${doc.id}/` })) } }, async () => { const posts = await getCollection('blog') return { title: 'Latest Posts', links: posts .filter(p => !p.data.draft) .slice(0, 5) .map(p => ({ title: p.data.title, url: `/blog/${p.id}/`, description: p.data.description })) } } ] } ``` -------------------------------- ### Changefreq Rule Usage Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/04-sitemap-api.md Example of how to configure change frequency rules. Sets the update frequency for different URL patterns. ```typescript { changefreq: [ { pattern: '/blog/', changefreq: 'weekly' }, { pattern: '/static/', changefreq: 'never' }, { pattern: /^\/products\//, changefreq: 'daily' } ] } ``` -------------------------------- ### LlmsLink Usage Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/03-llms-api.md Provides an example of a LlmsLink object with a title, URL, and description, illustrating its rendered format. ```typescript { title: 'Getting started', url: '/docs/getting-started', description: 'Setup and configuration guide' } // Renders: - [Getting started](/docs/getting-started): Setup and configuration guide ``` -------------------------------- ### Example RegistryBot Entry Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/09-registry-api.md An example of a `RegistryBot` object, demonstrating the expected format for properties like ID, provider, user agents, categories, and verification status. ```typescript { id: 'GPTBot', provider: 'OpenAI', userAgents: ['GPTBot'], categories: ['ai-training'], verified: true } ``` -------------------------------- ### Team Data Structure Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/06-security-humans-api.md Provides an example of how to structure an array of HumansTeamMember objects for use with the rendering function. ```typescript const team: HumansTeamMember[] = [ { name: 'Alice Chen', role: 'Lead Developer', location: 'San Francisco', email: 'alice@example.com' }, { name: 'Bob Johnson', role: 'Designer', twitter: '@bobjohn', location: 'New York' }, { name: 'Charlie Lee', role: 'Product Manager' } ] ``` -------------------------------- ### Generated security.txt output Source: https://github.com/casoon/astro-site-files/blob/main/README.md Example of the generated security.txt file content based on the provided configuration. ```text Contact: mailto:info@casoon.de Expires: 2027-01-01T00:00:00.000Z Acknowledgments: https://www.casoon.de/hall-of-fame Preferred-Languages: en, de Policy: https://www.casoon.de/security-policy Hiring: https://www.casoon.de/jobs ``` -------------------------------- ### Build-time RSS Feed Generation Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/05-rss-api.md Configures an RSS feed using the siteFiles function. The getItems function runs during the build process and reads local files to generate feed content. ```typescript siteFiles({ sitemap: { rss: { title: 'My Blog', description: 'Latest articles', language: 'en', getItems: async (siteUrl) => { // Runs in astro:build:done context // Use filesystem reads or imports, not getCollection() const { readFileSync, readdirSync } = await import('node:fs') const matter = (await import('gray-matter')).default const posts = readdirSync('./src/content/blog') .filter(f => f.endsWith('.mdx')) .map(file => { const { data } = matter(readFileSync(`./src/content/blog/${file}`, 'utf-8')) if (data.draft) return null return { title: data.title, pubDate: data.date, link: `${siteUrl}/blog/${file.replace(/\.mdx$/, '')}/`, description: data.description } }) .filter(Boolean) .sort((a, b) => new Date(b.pubDate) - new Date(a.pubDate)) return posts } } } }) ``` -------------------------------- ### Example Sitemap Audit Issue: No Site URL Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/07-audit-api.md An example `AuditIssue` for the `sitemap/no-site-url` rule, indicating that the site URL is not configured, which will result in relative paths in the sitemap. ```json { level: 'warn', rule: 'sitemap/no-site-url', message: 'No site URL configured — sitemap entries will be relative paths only.', help: 'Set `site` in astro.config.ts or pass `sitemap.siteUrl` to produce absolute URLs.' } ``` -------------------------------- ### Astro Configuration with siteFiles Integration Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/01-core-integration.md Example of how to configure the `siteFiles` integration within an `astro.config.ts` file. Ensure your `site` is defined in the Astro config for correct meta-file generation. ```typescript // astro.config.ts import { defineConfig } from 'astro/config' import siteFiles from '@casoon/astro-site-files' export default defineConfig({ site: 'https://example.com', integrations: [ siteFiles({ robots: { preset: 'seoOnly' }, llms: { title: 'My Site', description: 'A site.' }, sitemap: { exclude: ['/admin/'] }, security: { contact: 'mailto:security@example.com' }, humans: { team: [{ name: 'Alice', role: 'Dev' }], technology: ['Astro'] }, }) ] }) ``` -------------------------------- ### Example Sitemap Audit Issue: Empty Sitemap Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/07-audit-api.md An example `AuditIssue` for the `sitemap/empty-sitemap` rule, indicating that the sitemap has no entries after filtering. ```json { level: 'warn', rule: 'sitemap/empty-sitemap', message: 'Sitemap has no entries.', help: 'Verify your build produced pages or that `sitemap.sources` returns data.' } ``` -------------------------------- ### Usage Example: Robots.txt Generation Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/02-robots-api.md Demonstrates how to use `renderRobotsTxt` with presets, manual configuration including per-bot overrides, and explicit agent rules. Ensure the necessary import is included. ```typescript import { renderRobotsTxt } from '@casoon/astro-site-files' // Using a preset const txt1 = renderRobotsTxt( { preset: 'seoOnly' }, 'https://example.com' ) // Manual configuration with per-bot overrides const txt2 = renderRobotsTxt( { disallow: ['/admin/', '/private/'], allow: ['/admin/public/'], crawlDelay: 2, bots: { PerplexityBot: 'disallow' } }, 'https://example.com' ) // Without preset, explicit rules only const txt3 = renderRobotsTxt( { disallow: ['/cgi-bin/', '/tmp/'], agents: [ { userAgent: 'Googlebot', crawlDelay: 1 } ] } ) ``` -------------------------------- ### robots.txt with Custom Bots Source: https://github.com/casoon/astro-site-files/blob/main/README.md Add custom bots to the registry with specific user agents and categories. This example defines 'MyBot' for AI training. ```ts siteFiles({ robots: { preset: 'seoOnly', extraBots: [ { id: 'MyBot', provider: 'Example', userAgents: ['MyBot/1.0'], categories: ['ai-training'], verified: false } ] } }) ``` -------------------------------- ### Generated humans.txt Output Example Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/06-security-humans-api.md This is an example of the output generated by the renderHumansTxt function based on the provided options. ```plaintext /* TEAM */ Name: Alice Role: Development Location: Berlin Name: Bob Role: Design Twitter: @bob /* THANKS */ Open Source Community Our early users /* SITE LAST UPDATED */ 2026-06-24 /* TECHNOLOGY COLOPHON */ Astro, TypeScript, Tailwind CSS /* NOTE */ Built with care. ``` -------------------------------- ### Example Humans.txt Audit Issue Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/07-audit-api.md This is an example of an `AuditIssue` object that might be returned by `auditHumans` when no team entries are found in humans.txt. ```json { level: 'info', rule: 'humans/no-team', message: 'humans.txt has no team entries', help: 'Add a `team` array with at least one member to credit the people behind the site.' } ``` -------------------------------- ### Integration with Astro Configuration Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/03-llms-api.md Shows how to integrate the `siteFiles` function with LLM configuration within an Astro project's integration setup. Sources are resolved at build time. ```typescript siteFiles({ llms: { title: 'My Site', description: 'Blog and docs.', sources: [ async () => { const posts = await getCollection('blog') return { title: 'Blog', links: posts.map(p => ({ title: p.data.title, url: `/blog/${p.id}/` })) } } ] } }) ``` -------------------------------- ### Generated humans.txt output Source: https://github.com/casoon/astro-site-files/blob/main/README.md Example of the generated humans.txt file content, including team information, last update, and technologies. ```text /* TEAM */ Name: Alice Role: Development Location: Berlin /* SITE LAST UPDATED */ 2026-05-06 /* TECHNOLOGY COLOPHON */ Astro, TypeScript, Tailwind CSS ``` -------------------------------- ### Example LLM Audit Issue: No Sections Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/07-audit-api.md Shows an 'info' level audit issue when llms.txt is missing section configurations. This example helps in identifying and resolving issues related to structured content access for AI models. ```typescript { level: 'info', rule: 'llms/no-sections', message: 'llms.txt has no sections', help: 'Add sections with links to key pages — contact, services, team, location, working method. This gives AI models structured access to your most important content.' } ``` -------------------------------- ### Configure Sitemap Generation Source: https://github.com/casoon/astro-site-files/blob/main/README.md Configure sitemap generation by specifying exclusions, priority rules, and dynamic sources. This example excludes '/landing/' and sets a priority for '/blog/' paths, while dynamically fetching blog posts to include in the sitemap. ```typescript siteFiles({ sitemap: { exclude: ['/landing/'], priority: [{ pattern: '/blog/', priority: 0.9 }], sources: [ async () => { const posts = await getCollection('blog') return posts.map(p => ({ loc: `/blog/${p.id}/`, lastmod: p.data.date })) } ] } }) ``` -------------------------------- ### Build-time RSS Feed Rendering Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Example of using `renderRssFeed` to generate an RSS feed during the build process, providing feed metadata and a list of items. ```typescript import { renderRssFeed } from '@casoon/astro-site-files' const feed = renderRssFeed( { title: 'My Blog', description: 'Latest posts', language: 'en' }, 'https://example.com', [ { title: 'Post 1', pubDate: new Date(), link: '/blog/post-1/', description: 'First post' } ] ) ``` -------------------------------- ### Access and Filter Default Registry Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/09-registry-api.md Demonstrates how to import and use the `defaultRegistry` to view the total number of bots, find a specific bot by ID, and filter bots based on their categories. Ensure the `@casoon/astro-site-files` package is installed. ```typescript import { defaultRegistry } from '@casoon/astro-site-files' // View all known bots console.log(defaultRegistry.length) // 27 // Find a specific bot const openai = defaultRegistry.find(b => b.id === 'GPTBot') console.log(openai.categories) // ['ai-training'] // List all AI bots const aiBot = defaultRegistry.filter(b => b.categories.some(c => c.startsWith('ai-')) ) ``` -------------------------------- ### Bot Resolution Example: GPTBot Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/09-registry-api.md Demonstrates the resolution flow for GPTBot, a verified AI bot with the 'ai-training' category, when using the 'seoOnly' preset. The resolution is 'disallow' due to `groups.verifiedAi`. ```typescript // GPTBot (verified, ai-training category) // With preset: 'seoOnly' → groups.verifiedAi: 'disallow' // Resolution: disallow ``` -------------------------------- ### Importing Main and RSS Sub-module Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Demonstrates how to import from the main entry point and the RSS sub-module using conditional exports. ```javascript import { renderRobotsTxt } from '@casoon/astro-site-files' ``` ```javascript import { createRssRoute } from '@casoon/astro-site-files/rss' ``` -------------------------------- ### Example Sitemap Audit Issue: Duplicate URLs Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/07-audit-api.md An example `AuditIssue` for the `sitemap/duplicate-urls` rule, which can be a warning or an error depending on configuration. It indicates duplicate URLs found in the sitemap. ```json { level: 'warn', // or 'error' if errorOnDuplicates: true rule: 'sitemap/duplicate-urls', message: '2 duplicate URL(s) found: https://example.com/blog/, https://example.com/page/ … (+0 more). Duplicates are removed (last wins).', help: 'Check `sitemap.sources` and static routes for overlapping URLs. Add `sitemap/duplicate-urls` to `audit.disable` to suppress.' } ``` -------------------------------- ### Example LLM Audit Issue Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/07-audit-api.md Illustrates an 'info' level audit issue for a missing description in llms.txt. This example is useful for understanding the structure of issues reported by the `auditLlms` function. ```typescript { level: 'info', rule: 'llms/no-description', message: 'llms.txt has no description', help: 'Add a short summary of your site or company under `description`. AI models use this to understand your content at a glance.' } ``` -------------------------------- ### Example Security Audit Issue: No Expires Field Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/07-audit-api.md Demonstrates a 'warn' level audit issue indicating the absence of an 'Expires' field in security.txt, which is required by RFC 9116. This example is crucial for ensuring compliance with security.txt standards. ```typescript { level: 'warn', rule: 'security/no-expires', message: 'security.txt has no Expires field', help: 'RFC 9116 requires an Expires date so that outdated contact information is not trusted. Add `expires` as an ISO 8601 date string, e.g. "2027-01-01T00:00:00.000Z".' } ``` -------------------------------- ### Define LLM Metadata with Manual Sections Source: https://github.com/casoon/astro-site-files/blob/main/README.md Use `siteFiles` to configure LLM metadata, including title, description, details, and manually defined sections with links. ```typescript siteFiles({ llms: { title: 'Example', description: 'An example website focused on TypeScript tooling.', details: 'This site documents internal tools and workflows.', sections: [ { title: 'Documentation', links: [ { title: 'Getting started', url: '/docs/start', description: 'Setup guide' }, { title: 'API reference', url: '/docs/api' } ] } ] } }) ``` -------------------------------- ### Configure security.txt Source: https://github.com/casoon/astro-site-files/blob/main/README.md Configure the security.txt file with contact information, policy URLs, and other relevant details. The contact field is required. ```typescript siteFiles({ security: { contact: 'mailto:info@casoon.de', policy: 'https://www.casoon.de/security-policy', acknowledgments: 'https://www.casoon.de/hall-of-fame', preferredLanguages: ['en', 'de'], expires: '2027-01-01T00:00:00.000Z', hiring: 'https://www.casoon.de/jobs' } }) ``` -------------------------------- ### Programmatic Rendering of Site Files Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Shows how to programmatically render various site files like robots.txt, LLMs.txt, security.txt, humans.txt, and sitemaps using exported functions. ```typescript import { renderRobotsTxt, renderLlmsTxt, renderSecurityTxt, renderHumansTxt, renderSitemapXml, renderSitemapIndex, resolveEntry, deduplicateEntries, defaultRegistry, REGISTRY_VERSION, } from '@casoon/astro-site-files' // Robots const robots = renderRobotsTxt({ preset: 'seoOnly' }, 'https://example.com') // LLMs const llms = renderLlmsTxt({ title: 'My Site', description: 'A site.' }) // Security const security = renderSecurityTxt({ contact: 'mailto:sec@example.com' }) // Humans const humans = renderHumansTxt({ team: [{ name: 'Alice' }], technology: ['Astro'] }) // Sitemap const entries = [ { loc: '/page/' }, { loc: 'https://example.com/blog/', priority: 0.9 } ].map(e => resolveEntry(e, {}, 'https://example.com')) const deduped = deduplicateEntries(entries) const sitemap = renderSitemapXml(deduped) ``` -------------------------------- ### SiteFilesOptions Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/08-types-reference.md Complete configuration object passed to the `siteFiles()` integration. This object allows customization of robots.txt, llms.txt, sitemap.xml, security.txt, humans.txt, and audit settings. ```APIDOC ## SiteFilesOptions ### Description Configuration object for the `siteFiles()` integration, controlling various site file generation options. ### Fields - **robots** (`RobotsOptions | boolean`) - Optional - robots.txt config. `false` to disable. - **llms** (`LlmsOptions | boolean`) - Optional - llms.txt config. Omit to skip. - **sitemap** (`SitemapOptions | boolean`) - Optional - sitemap.xml config. `false` to disable. - **security** (`SecurityOptions | boolean`) - Optional - security.txt config. Omit to skip. - **humans** (`HumansOptions | boolean`) - Optional - humans.txt config. Omit to skip. - **audit** (`AuditOptions | boolean`) - Optional - Build-time hints. Set `false` to silence. - **debug** (`boolean`) - Optional - Log internal details to console. ``` -------------------------------- ### RSS Feed XML Structure Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/05-rss-api.md This is an example of the generated RSS 2.0 XML structure. It includes standard channel elements and item details. ```xml ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ``` -------------------------------- ### Basic Integration in Astro Config Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/00-index.md Configure the Astro Site Files integration directly in your `astro.config.ts` file. This is the most common way to set up features like robots.txt, LLM metadata, sitemaps, security contact, and humans.txt. ```typescript import siteFiles from '@casoon/astro-site-files' export default defineConfig({ site: 'https://example.com', integrations: [ siteFiles({ robots: { preset: 'seoOnly' }, llms: { title: 'My Site', description: '...' }, sitemap: { exclude: ['/admin/'] }, security: { contact: 'mailto:security@example.com' }, humans: { team: [{ name: 'Alice' }], technology: ['Astro'] }, }) ] }) ``` -------------------------------- ### robots.txt with Per-Bot Override Source: https://github.com/casoon/astro-site-files/blob/main/README.md Apply the highest precedence override by specifying rules for individual bots. This example blocks 'PerplexityBot' even if the preset allows it. ```ts siteFiles({ robots: { preset: 'blockTraining', bots: { PerplexityBot: 'disallow' } // also block AI search } }) ``` -------------------------------- ### robots.txt with Preset Only Source: https://github.com/casoon/astro-site-files/blob/main/README.md Apply a predefined preset like 'seoOnly' for a quick configuration. Presets manage group defaults and known-bot rules. ```ts siteFiles({ robots: { preset: 'seoOnly' } }) ``` -------------------------------- ### Basic robots.txt with Preset and Disallow Source: https://github.com/casoon/astro-site-files/blob/main/README.md Use a preset like 'seoOnly' to block AI training and archiving while allowing search engines. Add specific paths to disallow for all user agents. ```ts siteFiles({ robots: { preset: 'seoOnly', // blocks AI training, archiving; search engines allowed disallow: ['/admin'], // additional paths for User-agent: * } }) ``` -------------------------------- ### Bot Resolution Example: Custom Bot Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/09-registry-api.md Shows the resolution for a custom bot that has a `defaultAction` of 'disallow' and no preset is applied. The resolution is 'disallow' based on its `defaultAction`. ```typescript // Custom bot with no preset // defaultAction: 'disallow' // Resolution: disallow ``` -------------------------------- ### Render Robots.txt with Preset Source: https://github.com/casoon/astro-site-files/blob/main/README.md Generates robots.txt using a predefined 'seoOnly' preset. Requires the base URL of the site. ```typescript import { renderRobotsTxt, } from '@casoon/astro-site-files' // With preset const robots = renderRobotsTxt({ preset: 'seoOnly' }, 'https://example.com') ``` -------------------------------- ### Bot Resolution Example: Diffbot Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/09-registry-api.md Illustrates the resolution for Diffbot, an unverified bot in the 'unknown-ai' category, when using the 'seoOnly' preset. The resolution is 'disallow' based on `groups.unknownAi`. ```typescript // Diffbot (unverified, unknown-ai category) // With preset: 'seoOnly' → groups.unknownAi: 'disallow' // Resolution: disallow ``` -------------------------------- ### Programmatic Rendering - renderRobotsTxt Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Renders the content for a robots.txt file. ```APIDOC ## renderRobotsTxt ### Description Generates the content for a `robots.txt` file. ### Method Not applicable (function export) ### Endpoint Not applicable (function export) ### Parameters #### Request Body - **options** (object) - Optional - Configuration options for `robots.txt`, such as `preset`. - **siteUrl** (string) - Required - The base URL of the site. ### Request Example ```ts import { renderRobotsTxt } from '@casoon/astro-site-files' const robots = renderRobotsTxt({ preset: 'seoOnly' }, 'https://example.com') ``` ### Response - **string** - The content for the `robots.txt` file. ``` -------------------------------- ### robots.txt with Group Override Source: https://github.com/casoon/astro-site-files/blob/main/README.md Override a specific group's behavior within a preset. This example blocks SEO scanners in addition to the 'seoOnly' preset's rules. ```ts siteFiles({ robots: { preset: 'seoOnly', groups: { seoScanners: 'disallow' } // also block SEO scanners } }) ``` -------------------------------- ### siteFiles() Integration Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/01-core-integration.md The siteFiles() function is an Astro integration that generates various site meta-files. It accepts an optional configuration object to customize the output of files like robots.txt, llms.txt, sitemap.xml, security.txt, and humans.txt. ```APIDOC ## siteFiles() ### Description Astro integration that generates site meta-files (`robots.txt`, `llms.txt`, `sitemap.xml`, `security.txt`, `humans.txt`) at build time. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Method This is an Astro integration, not an HTTP endpoint. It is invoked during the Astro build process. ### Endpoint N/A (Astro Integration) ### Parameters #### Options - **options** (`SiteFilesOptions`) - Optional - Configuration object for all generated files. ### Return Type `AstroIntegration` — An Astro-compatible integration object. ### Usage Example ```ts // astro.config.ts import { defineConfig } from 'astro/config' import siteFiles from '@casoon/astro-site-files' export default defineConfig({ site: 'https://example.com', integrations: [ siteFiles({ robots: { preset: 'seoOnly' }, llms: { title: 'My Site', description: 'A site.' }, sitemap: { exclude: ['/admin/'] }, security: { contact: 'mailto:security@example.com' }, humans: { team: [{ name: 'Alice', role: 'Dev' }], technology: ['Astro'] }, }) ] }) ``` ### Configuration See `SiteFilesOptions` in [types.md](types.md) for all available options. ``` -------------------------------- ### Configure @casoon/astro-site-files in astro.config.ts Source: https://github.com/casoon/astro-site-files/blob/main/README.md Integrate the package into your Astro configuration. Set the site URL and configure individual file generation options like robots, llms, and security. ```ts import { defineConfig } from 'astro/config' import siteFiles from '@casoon/astro-site-files' export default defineConfig({ site: 'https://example.com', integrations: [ siteFiles({ robots: { preset: 'seoOnly', // blocks AI training and archives; search engines stay allowed disallow: ['/admin'], // additional path rules on top of the preset }, llms: { title: 'Example', description: 'An example website.' }, security: { contact: 'mailto:info@casoon.de' }, humans: { team: [{ name: 'Alice', role: 'Development' }], technology: ['Astro', 'TypeScript'] } }) ] }) ``` -------------------------------- ### RobotsOptions Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/08-types-reference.md Options for configuring the robots.txt file, including disallow/allow rules, sitemap directive, crawl delay, presets, and custom bot configurations. ```APIDOC ## RobotsOptions ### Description Configuration options for generating the robots.txt file. ### Fields - **disallow** (`string[]`) - Optional - Paths to disallow for `User-agent: *`. - **allow** (`string[]`) - Optional - Paths to allow for `User-agent: *`. - **sitemap** (`boolean | string`) - Optional - Sitemap directive. `true` = auto, `string` = explicit URL, `false` = omit. - **crawlDelay** (`number`) - Optional - Crawl-delay seconds for `User-agent: *`. - **preset** (`Preset`) - Optional - Named preset (`seoOnly`, `citationFriendly`, `openToAi`, `blockTraining`, `lockdown`). - **bots** (`Record`) - Optional - Per-bot action overrides (highest priority). - **groups** (`Groups`) - Optional - Group-level action controls (override preset defaults). - **extraBots** (`RegistryBot[]`) - Optional - Custom bots to merge into registry. - **agents** (`AgentRule[]`) - Optional - Explicit per-agent rule blocks. ``` -------------------------------- ### Get Registry Version Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/09-registry-api.md Access the constant `REGISTRY_VERSION` to retrieve the ISO date string indicating the last registry update. Useful for logging, debugging, and displaying the registry update time. ```typescript export const REGISTRY_VERSION: string ``` ```typescript import { REGISTRY_VERSION } from '@casoon/astro-site-files' console.log(REGISTRY_VERSION) // '2026-05-27' ``` -------------------------------- ### Auditing Site Files Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Shows how to use audit functions like `auditRobots` and `auditSecurity` to check site files for issues. It also demonstrates filtering issues based on options. ```typescript import { auditRobots, auditLlms, auditSecurity, auditHumans, auditSitemap, filterIssues, type AuditIssue, type AuditOptions, } from '@casoon/astro-site-files' const robotsIssues = auditRobots({ disallow: ['/admin'] }) const securityIssues = auditSecurity({ contact: 'mailto:sec@example.com' }) const config: AuditOptions = { disable: ['llms/no-description'] } const filtered = filterIssues([...robotsIssues, ...securityIssues], config) for (const issue of filtered) { console.log(`[${issue.rule}] ${issue.message}`) } ``` -------------------------------- ### Filter Verified AI Bots and Training Bots Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/09-registry-api.md Demonstrates how to filter the default registry to get verified AI bots and bots used for training. Useful for creating custom bot management rules. ```typescript import { defaultRegistry } from '@casoon/astro-site-files' // Get all verified AI bots const verifiedAi = defaultRegistry.filter(b => b.verified && b.categories.some(c => c.startsWith('ai-')) ) // Get all training bots const trainingBots = defaultRegistry.filter(b => b.categories.includes('ai-training') ) // Create a custom group const toBlock = trainingBots.map(b => b.id) console.log('Bots to block for training:', toBlock) // Output: ['GPTBot', 'ClaudeBot', 'anthropic-ai', 'Google-Extended', 'CCBot', 'Bytespider', 'Applebot-Extended'] ``` -------------------------------- ### Programmatic Rendering - renderSecurityTxt Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Renders the content for a security.txt file. ```APIDOC ## renderSecurityTxt ### Description Generates the content for a `security.txt` file. ### Method Not applicable (function export) ### Endpoint Not applicable (function export) ### Parameters #### Request Body - **options** (object) - Required - Configuration options for `security.txt`, such as `contact`. ### Request Example ```ts import { renderSecurityTxt } from '@casoon/astro-site-files' const security = renderSecurityTxt({ contact: 'mailto:sec@example.com' }) ``` ### Response - **string** - The content for the `security.txt` file. ``` -------------------------------- ### Configure Build-Time RSS Feed Generation Source: https://github.com/casoon/astro-site-files/blob/main/README.md Use this configuration in `sitemap.rss` to generate an `rss.xml` file during the Astro build process. The `getItems` function should use filesystem reads as it runs in `astro:build:done` context. ```typescript siteFiles({ sitemap: { rss: { title: 'My Blog', description: 'Latest articles about TypeScript and Astro.', language: 'en', getItems: async (siteUrl) => { const { readdirSync, readFileSync } = await import('node:fs') const matter = (await import('gray-matter')).default const dir = './src/content/blog' return readdirSync(dir) .filter(f => f.endsWith('.mdx')) .map(file => { const { data } = matter(readFileSync(`${dir}/${file}`, 'utf-8')) if (data.draft) return null return { title: data.title, pubDate: data.date, link: `${siteUrl}/blog/${file.replace(/\.mdx$/, '')}/`, description: data.description, } }) .filter(Boolean) .sort((a, b) => new Date(b.pubDate) - new Date(a.pubDate)) }, }, }, }) ``` -------------------------------- ### Programmatic Rendering - renderLlmsTxt Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Renders the content for a LLMs.txt file. ```APIDOC ## renderLlmsTxt ### Description Generates the content for a `LLMs.txt` file. ### Method Not applicable (function export) ### Endpoint Not applicable (function export) ### Parameters #### Request Body - **options** (object) - Required - Configuration options for `LLMs.txt`, such as `title` and `description`. ### Request Example ```ts import { renderLlmsTxt } from '@casoon/astro-site-files' const llms = renderLlmsTxt({ title: 'My Site', description: 'A site.' }) ``` ### Response - **string** - The content for the `LLMs.txt` file. ``` -------------------------------- ### Importing Type Definitions Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Shows how to import all public type definitions from the main entry point for use in TypeScript projects. ```typescript import type { SiteFilesOptions, RobotsOptions, LlmsOptions, SecurityOptions, HumansOptions, SitemapOptions, RssItem, AuditIssue, AuditOptions, } from '@casoon/astro-site-files' ``` -------------------------------- ### Configure humans.txt Source: https://github.com/casoon/astro-site-files/blob/main/README.md Configure the humans.txt file to list team members, acknowledgments, technologies used, and a personal note. The team field accepts an array of member objects. ```typescript siteFiles({ humans: { team: [ { name: 'Alice', role: 'Development', location: 'Berlin' }, { name: 'Bob', role: 'Design', twitter: '@bob' } ], thanks: ['Open Source Community', 'Our early users'], technology: ['Astro', 'TypeScript', 'Tailwind CSS'], note: 'Built with care.' } }) ``` -------------------------------- ### Programmatic Rendering - renderSitemapIndex Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md Renders the content for a sitemap index file. ```APIDOC ## renderSitemapIndex ### Description Generates the content for a sitemap index file. ### Method Not applicable (function export) ### Endpoint Not applicable (function export) ### Parameters #### Request Body - **entries** (Array) - Required - An array of resolved sitemap entries, typically pointing to individual sitemap files. ### Request Example ```ts import { renderSitemapIndex } from '@casoon/astro-site-files' // Assuming 'entries' is an array of sitemap file locations const sitemapIndex = renderSitemapIndex(entries) ``` ### Response - **string** - The rendered sitemap index in XML format. ``` -------------------------------- ### SitemapOptions Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/08-types-reference.md Configuration options for generating the sitemap. ```APIDOC ## SitemapOptions ### Description Configuration options for generating the sitemap. ### Fields - **siteUrl** (`string`) - Optional - Override site URL (defaults to `astro.config.site`) - **sources** (`SitemapSource[]`) - Optional - Async entry providers (defaults to `[]`) - **exclude** (`(string | RegExp)[]`) - Optional - URL patterns to exclude (defaults to `[]`) - **filter** (`function`) - Optional - Custom absolute URL filter - **priority** (`PriorityRule[]`) - Optional - Priority overrides (defaults to `[]`) - **changefreq** (`ChangefreqRule[]`) - Optional - Changefreq overrides (defaults to `[]`) - **output.mode** (`'single' | 'index'`) - Optional - Output format (defaults to `'single'`) - **output.maxUrls** (`number`) - Optional - URLs per chunk (index mode) (defaults to `50000`) - **output.filename** (`string`) - Optional - Filename (single mode) (defaults to `'sitemap.xml'`) - **audit.warnOnEmpty** (`boolean`) - Optional - Warn if no entries (defaults to `true`) - **audit.errorOnDuplicates** (`boolean`) - Optional - Error on duplicates (defaults to `false`) - **serialize** (`function`) - Optional - Per-entry transform hook - **i18n** (`I18nOptions`) - Optional - i18n configuration - **rss** (`RssConfig`) - Optional - RSS feed config - **debug** (`boolean`) - Optional - Debug logging (defaults to `false`) ``` -------------------------------- ### Programmatic Rendering - REGISTRY_VERSION Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/10-module-exports.md The version of the default registry. ```APIDOC ## REGISTRY_VERSION ### Description Indicates the version of the default registry. ### Method Not applicable (constant export) ### Endpoint Not applicable (constant export) ### Parameters None. ### Request Example ```ts import { REGISTRY_VERSION } from '@casoon/astro-site-files' console.log(REGISTRY_VERSION) ``` ### Response - **string** - The registry version. ``` -------------------------------- ### renderSitemapIndex() Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/04-sitemap-api.md Generates a sitemap index file (`sitemap-index.xml`) that references multiple sitemap chunks. This is useful when the total number of URLs exceeds the limit for a single sitemap file. ```APIDOC ## renderSitemapIndex() ### Description Generates a `sitemap-index.xml` file that references multiple sitemap chunks. Used when the sitemap exceeds the URL limit (default 50,000 per XML file). ### Parameters | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `sitemaps` | `Array<{ loc: string; lastmod: string }>` | Yes | Array of sitemap file references | ### Return Type `string` — Complete `sitemap-index.xml` file content as valid XML. ### Usage Example ```ts import { renderSitemapIndex } from '@casoon/astro-site-files' const index = renderSitemapIndex([ { loc: 'https://example.com/sitemap-1.xml', lastmod: '2026-06-24' }, { loc: 'https://example.com/sitemap-2.xml', lastmod: '2026-06-24' }, { loc: 'https://example.com/sitemap-3.xml', lastmod: '2026-06-24' } ]) ``` ### XML Structure ```xml ... ... ... ``` ``` -------------------------------- ### siteFiles Integration Source: https://github.com/casoon/astro-site-files/blob/main/_autodocs/00-index.md The main integration function to set up Astro Site Files in your Astro project. ```APIDOC ## siteFiles ### Description Main integration function for Astro Site Files. ### Signature `siteFiles(options?: SiteFilesOptions): AstroIntegration` ### Parameters #### Options - **options** (SiteFilesOptions) - Optional - Configuration options for the integration. ```