/llms.txt file

https://llmstxt.org/
Admin
A proposal to standardize on using an /llms.txt file on websites to provide concise, LLM-friendly...
Tokens:7,319
Snippets:99
Trust Score:9.3
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install the library

Source: https://llmstxt.org/intro.html

Install the package using pip.

```sh
pip install llms-txt
```

--------------------------------

### Adding resource sections to llms.txt

Source: https://llmstxt.org/nbdev.md

Example of how to organize documentation and example links using H2 headings.

```markdown
## Docs

- [Surreal](https://host/README.md): Tiny jQuery alternative with Locality of Behavior
- [FastHTML quick start](https://host/quickstart.html.md): An overview of FastHTML features

## Examples

- [Todo app](https://host/adv_app.py)
```

--------------------------------

### Example library description for llms.txt

Source: https://llmstxt.org/nbdev.md

A sample description section providing usage tips for a library.

```markdown
Here are some tips on using fastcore:

- **Liberal imports**: Utilize `from fastcore.module import *` freely. The library is designed for safe wildcard imports.
- **Enhanced list operations**: Substitute `list` with `L`. This provides advanced indexing, method chaining, and additional functionality while maintaining list-like behavior.
- **Extend existing classes**: Apply the `@patch` decorator to add methods to classes, including built-ins, without subclassing. This enables more flexible code organization.
```

--------------------------------

### Install llms-txt package

Source: https://llmstxt.org/intro.html.md

Install the package using pip.

```sh
pip install llms-txt
```

--------------------------------

### Start ed and enable help

Source: https://llmstxt.org/ed.md

Initializes the ed editor and enables helpful error messages.

```sh
$ ed
* H
```

--------------------------------

### Example Restaurant llms.txt

Source: https://llmstxt.org/domains.md

A sample structure for a restaurant's llms.txt file, including business hours and links to menu documentation.

```markdown
# Nate the Great's Grill

> Nate the Great's Grill is a popular destination off of Sesame Street that has been serving the community for over 20 years. We offer the best BBQ for a great price.

Here are our weekly hours:

- Monday - Friday: 9am - 9pm
- Saturday: 11am - 9pm
- Sunday: Closed

## Menus

- [Lunch Menu](https://host/lunch.html.md): Our lunch menu served from 11am to 4pm every day.
- [Dinner Menu](https://host/dinner.html.md): Our dinner menu served from 4pm to 9pm every day.

## Optional

- [Dessert Menu](https://host/dessert.md): A subset of the Starlette docs
```

--------------------------------

### Example Restaurant Menu Content

Source: https://llmstxt.org/domains.md

A sample markdown-formatted menu using tables to display items and prices.

```markdown
## By The Pound

| Item              | Price         |
| --------------    | -----------   |
| Brisket           | 34            |
| Pork Spare Ribs   | 30            |
| Pulled Pork       | 28            |

## Drinks

| Item              | Price         |
| --------------    | -----------   |
| Iced Tea          | 3             |
| Mexican Coke      | 3             |

## Sides

| Item              | Price         |
| --------------    | -----------   |
| Potato Salad      | 4             |
| Slaw              | 4             |
```

--------------------------------

### FastHTML llms.txt structure

Source: https://llmstxt.org/index.md

A sample llms.txt file demonstrating how to organize project summaries, documentation links, and examples for LLM consumption.

```markdown
# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.

Important notes:

- Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services
- FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte.

## Docs

- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options

## Examples

- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.

## Optional

- [Starlette full documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development. 
```

--------------------------------

### Example API Reference Excerpt

Source: https://llmstxt.org/nbdev.md

A sample of the Markdown output generated by pysym2md for a module.

```markdown
# fastcore Module Documentation

## fastcore.ansi

> Filters for processing ANSI colors.

- `def strip_ansi(source)`
    Remove ANSI escape codes from text.

- `def ansi2html(text)`
    Convert ANSI colors to HTML colors.

- `def ansi2latex(text)`
    Convert ANSI colors to LaTeX colors.
```

--------------------------------

### llms.txt Input Format

Source: https://llmstxt.org/llmstxt-js.html

Example of the markdown-based input structure for the llms.txt parser.

```markdown
# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)
```

--------------------------------

### Adding an Optional section to llms.txt

Source: https://llmstxt.org/nbdev.md

Example of an Optional section for secondary resources that agents can skip when context is limited.

```markdown
## Optional

- [Starlette docs](https://host/starlette-sml.md): A subset of the Starlette docs
```

--------------------------------

### JSON representation of parsed section

Source: https://llmstxt.org/intro.html.md

Example output structure for a parsed section.

```json
{ 'desc': 'A subset of the Starlette documentation useful for FastHTML '
          'development.',
  'title': 'Starlette full documentation',
  'url': 'https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md'}
```

--------------------------------

### Define and use mk_ctx function

Source: https://llmstxt.org/core.html

Creates a project context from a dictionary and converts it to XML.

```python
def mk_ctx(
    d, optional:bool=True, n_workers:NoneType=None
):
```

```python
ctx = mk_ctx(llmsd)
print(to_xml(ctx, do_escape=False)[:260]+'...')
```

--------------------------------

### Configure Quarto Build Process

Source: https://llmstxt.org/nbdev.md

Settings for nbs/_quarto.yml to automate API reference generation and include text files in the build.

```yaml
project:
  type: website
  pre-render:
    - pysym2md --output-file apilist.txt nbdev
  resources:
    - "*.txt"
```

--------------------------------

### Define create_ctx function

Source: https://llmstxt.org/core.html

Creates a project context from text, optionally skipping the optional section.

```python
def create_ctx(
    txt, optional:bool=False, n_workers:NoneType=None
):
```

--------------------------------

### Define and use llms_txt2ctx function

Source: https://llmstxt.org/core.html

Prints a project context from a file and provides a command-line interface for conversion.

```python
def llms_txt2ctx(
    fname:str, # File name to read
    optional:bool_arg=False, # Include 'optional' section?
    n_workers:int=None, # Number of threads to use for parallel downloading
    save_nbdev_fname:str=None, # save output to nbdev `{docs_path}` instead of emitting to stdout
):
```

```python
!llms_txt2ctx llms-sample.txt > ../fasthtml.md
```

--------------------------------

### Read and display llms.txt sample

Source: https://llmstxt.org/core.html

Reads the content of a local llms-sample.txt file and prints the first 480 characters.

```python
samp = Path('llms-sample.txt').read_text()
print(samp[:480])
```

--------------------------------

### create_ctx(txt, optional=False, n_workers=None)

Source: https://llmstxt.org/core.html

Generates a Project object from text input, creating a Section for each H2 header found.

```APIDOC
## create_ctx(txt, optional=False, n_workers=None)

### Description
A `Project` with a `Section` for each H2 part in `txt`, optionally skipping the ‘optional’ section.

### Parameters
- **txt** (str) - Required - The text content to parse.
- **optional** (bool) - Optional - Whether to include the 'optional' section. Defaults to False.
- **n_workers** (NoneType) - Optional - Number of workers for processing.
```

--------------------------------

### llms_txt2ctx(fname, optional=False, n_workers=None, save_nbdev_fname=None)

Source: https://llmstxt.org/core.html

Reads a file and prints a Project structure, optionally saving the output to a specific path.

```APIDOC
## llms_txt2ctx(fname, optional=False, n_workers=None, save_nbdev_fname=None)

### Description
Print a `Project` with a `Section` for each H2 part in file read from `fname`, optionally skipping the ‘optional’ section.

### Parameters
- **fname** (str) - Required - File name to read.
- **optional** (bool_arg) - Optional - Include 'optional' section? Defaults to False.
- **n_workers** (int) - Optional - Number of threads to use for parallel downloading.
- **save_nbdev_fname** (str) - Optional - Save output to nbdev `{docs_path}` instead of emitting to stdout.
```

--------------------------------

### Create LLM context file

Source: https://llmstxt.org/intro.html.md

Generate an XML context file from the parsed data.

```python
ctx = create_ctx(samp)
```

```python
print(ctx[:300])
```

--------------------------------

### Define and use get_sizes function

Source: https://llmstxt.org/core.html

Calculates the size of each section in the LLM context and writes the XML output to a file.

```python
def get_sizes(
    ctx
):
```

```python
get_sizes(ctx)
```

```python
Path('../fasthtml.md').write_text(to_xml(ctx, do_escape=False))
```

--------------------------------

### CLI usage for llms-txt

Source: https://llmstxt.org/intro.html.md

Commands for accessing help and converting llms.txt files to XML context.

```sh
llms_txt2ctx -h
```

```sh
llms_txt2ctx llms.txt > llms.md
```

--------------------------------

### Parse link components

Source: https://llmstxt.org/core.html

Demonstrates building regex patterns incrementally to extract titles, URLs, and descriptions from markdown-style links.

```python
link = '- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of FastHTML features'
```

```python
title = named_re('title', r'[^\]]+')
pat =  fr'-\s*\[{title}\]'
search(pat, samp)
```

```python
url = named_re('url', r'[^\)]+')
pat += fr'\({url}\)'
search(pat, samp)
```

```python
desc = named_re('desc', r'.*')
pat += opt_re(fr':\s*{desc}')
search(pat, link)
```

--------------------------------

### mk_ctx(d, optional=True, n_workers=None)

Source: https://llmstxt.org/core.html

Creates a Project object with a Section for each H2 part in the provided input, with options to filter sections.

```APIDOC
## mk_ctx(d, optional=True, n_workers=None)

### Description
Create a `Project` with a `Section` for each H2 part in `d`, optionally skipping the ‘optional’ section.

### Parameters
- **d** (object) - Required - The input data source.
- **optional** (bool) - Optional - Whether to include the 'optional' section. Defaults to True.
- **n_workers** (NoneType) - Optional - Number of workers for processing.
```

--------------------------------

### Load llms.txt context

Source: https://llmstxt.org/ed.md

Uses the hypothetical L command to retrieve and process llms.txt from a specified URL.

```sh
* L fastht.ml/docs
Checking for /llms.txt at fastht.ml/docs...
Found /llms.txt. Parsing...
Fetching URLs from "Docs" section...  Fetching URLs from "Examples" section...
Skipping "Optional" section for brevity.
Creating XML-based context for Claude...  Context created and loaded.
```

--------------------------------

### CLI usage

Source: https://llmstxt.org/intro.html

Commands for interacting with the llms_txt2ctx CLI tool.

```sh
llms_txt2ctx -h
```

```sh
llms_txt2ctx llms.txt > llms.md
```

--------------------------------

### Save and quit

Source: https://llmstxt.org/ed.md

Writes the buffer to a file and exits the editor.

```sh
*w hello_world.py
5
*q
```

--------------------------------

### get_sizes(ctx)

Source: https://llmstxt.org/core.html

Calculates and returns the size of each section within the provided LLM context.

```APIDOC
## get_sizes(ctx)

### Description
Get the size of each section of the LLM context.

### Parameters
- **ctx** (object) - Required - The context object to measure.
```

--------------------------------

### Execute AI generation

Source: https://llmstxt.org/ed.md

Uses the hypothetical x command to generate code based on a prompt and loaded context.

```sh
* x Create a simple FastHTML app which outputs 'Hello, World!', in a <div>.
Analyzing context and prompt...
Generating FastHTML app...
App written to buffer.
```

--------------------------------

### Define get_doc_content function

Source: https://llmstxt.org/core.html

Fetches content from a local file if within an nbdev repository.

```python
def get_doc_content(
    url
):
```

--------------------------------

### Implementation of llms.txt parser

Source: https://llmstxt.org/intro.html.md

A minimal implementation of an llms.txt parser using standard library modules.

```python
from pathlib import Path
import re,itertools

def chunked(it, chunk_sz):
    it = iter(it)
    return iter(lambda: list(itertools.islice(it, chunk_sz)), [])

def parse_llms_txt(txt):
    "Parse llms.txt file contents in `txt` to a `dict`"
    def _p(links):
        link_pat = '-\s*\[(?P<title>[^\]]+)\]\((?P<url>[^\)]+)\)(?::\s*(?P<desc>.*))?'
        return [re.search(link_pat, l).groupdict()
                for l in re.split(r'\n+', links.strip()) if l.strip()]

    start,*rest = re.split(fr'^##\s*(.*?$)', txt, flags=re.MULTILINE)
    sects = {k: _p(v) for k,v in dict(chunked(rest, 2)).items()}
    pat = '^#\s*(?P<title>.+?$)\n+(?:^>\s*(?P<summary>.+?$)$)?\n+(?P<info>.*)'
    d = re.search(pat, start.strip(), (re.MULTILINE|re.DOTALL)).groupdict()
    d['sections'] = sects
    return d
```

--------------------------------

### Generate API Reference via CLI

Source: https://llmstxt.org/nbdev.md

Command to generate an API reference for a library using pysym2md.

```bash
pysym2md numpy
```

--------------------------------

### Initial llms.txt structure

Source: https://llmstxt.org/nbdev.md

The basic structure of an llms.txt file, requiring an H1 header and a blockquote summary.

```markdown
# FastHTML

> FastHTML is a python library which...

When writing FastHTML apps remember to:

- Thing to remember
```

--------------------------------

### Verify and print buffer

Source: https://llmstxt.org/ed.md

Checks the line count and displays the contents of the current buffer.

```sh
* n
5
* p
from fasthtml.common import *
app,rt = fast_app()
@rt
def index(): return div("Hello, World!")
serve()
```

--------------------------------

### Test llms.txt with Claude

Source: https://llmstxt.org/domains.md

A Python script using the claudette library to fetch and provide llms.txt content as context for an interactive chat session.

```python
# /// script
# requires-python = ">=3.8"
# dependencies = [
#     "claudette",
#     "llms-txt",
#     "requests",
# ]
# ///
from claudette import *
from llms_txt import create_ctx

import requests

model = models[1] # Sonnet 3.5
chat = Chat(model, sp="""You are a helpful and concise assistant.""")

url = 'your_url/llms.txt'
text = requests.get(url).text
llm_ctx = create_ctx(text)
chat(llm_ctx + '\n\nThe above is necessary context for the conversation.')

while True:
    msg = input('Your question about the site: ')
    res = chat(msg)
    print('From Claude:', contents(res))
```

--------------------------------

### Configure Link Headers for LLM Discovery

Source: https://llmstxt.org/index.md

Use the Link HTTP header to point agents to markdown versions of pages and the relevant llms.txt file. This can be configured at the web server or CDN level without modifying HTML.

```http
Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"
```

--------------------------------

### Create optional regex match

Source: https://llmstxt.org/core.html

Generates a pattern to optionally match the provided string.

```python
def opt_re(
    s
):
```

--------------------------------

### Add Dependencies to settings.ini

Source: https://llmstxt.org/nbdev.md

Configuration to include pysym2md as a development requirement.

```ini
dev_requirements = pysym2md
```

--------------------------------

### Python API usage

Source: https://llmstxt.org/intro.html

Basic usage of the Python API to parse files and generate context.

```python
from llms_txt import *
```

```python
samp = Path('llms-sample.txt').read_text()
```

```python
parsed = parse_llms_file(samp)
list(parsed)
```

```python
parsed.title,parsed.summary
```

```python
list(parsed.sections)
```

```python
parsed.sections.Optional[0]
```

```python
ctx = create_ctx(samp)
```

```python
print(ctx[:300])
```

--------------------------------

### Parse link section

Source: https://llmstxt.org/core.html

Parses a full link section from an llms.txt file.

```python
def parse_link(
    txt
):
```

```python
parse_link(link)
```

```python
parse_link('-[foo](http://foo)')
```

--------------------------------

### get_doc_content(url)

Source: https://llmstxt.org/core.html

Fetches content from a local file if it is located within an nbdev repository.

```APIDOC
## get_doc_content(url)

### Description
Fetch content from local file if in nbdev repo.

### Parameters
- **url** (str) - Required - The URL or path to the document.
```

--------------------------------

### Search text with regex

Source: https://llmstxt.org/core.html

Returns a dictionary of matched groups for a pattern within the provided text.

```python
def search(
    pat, txt, flags:int=0
):
```

--------------------------------

### Parse document sections

Source: https://llmstxt.org/core.html

Splits document content by headers and extracts links within those sections.

```python
sections = '''First bit.

## S1

-[foo](http://foo)
- [foo2](http://foo2): stuff

## S2

- [foo3](http://foo3)'''
```

```python
start,*rest = re.split(fr'^##\s*(.*?$)', sections, flags=re.MULTILINE)
start
```

```python
rest
```

```python
d = dict(chunked(rest, 2))
d
```

```python
links = d['S1']
links.strip()
```

```python
_parse_links(links)
```

```python
start, sects = _parse_llms(samp)
start
```

```python
title = named_re('title', r'.+?$')
summ = named_re('summary', '.+?$')
summ_pat = opt_re(fr"^>\s*{summ}$")
info = named_re('info', '.*')
```

```python
pat = fr'^#\s*{title}\n+{summ_pat}\n+{info}'
search(pat, start, (re.MULTILINE|re.DOTALL))
```

--------------------------------

### Parse full llms.txt file

Source: https://llmstxt.org/core.html

Parses the entire contents of an llms.txt file into an AttrDict structure.

```python
def parse_llms_file(
    txt
):
```

```python
llmsd = parse_llms_file(samp)
llmsd.summary
```

```python
llmsd.sections.Examples
```

--------------------------------

### Python API for parsing llms.txt

Source: https://llmstxt.org/intro.html.md

Basic usage of the Python module to read and parse an llms.txt file.

```python
from llms_txt import *
```

```python
samp = Path('llms-sample.txt').read_text()
```

```python
parsed = parse_llms_file(samp)
list(parsed)
```

```python
parsed.title,parsed.summary
```

```python
list(parsed.sections)
```

```python
parsed.sections.Optional[0]
```

--------------------------------

### llms.txt Parser Output

Source: https://llmstxt.org/llmstxt-js.html

The resulting JSON structure produced by the parser from the provided input.

```json
{
  "title": "Title",
  "summary": "Optional description goes here",
  "info": "Optional details go here",
  "sections": {
    "Section name": [
      {
        "title": "Link title",
        "url": "https://link_url",
        "desc": "Optional link details"
      }
    ],
    "Optional": [
      {
        "title": "Link title",
        "url": "https://link_url"
      }
    ]
  }
}
```

--------------------------------

### Create named regex capture group

Source: https://llmstxt.org/core.html

Generates a pattern to match a specific pattern within a named capture group.

```python
def named_re(
    nm, pat
):
```
Tokens:
Raw