llms.txt

https://llmstxt.org/llms.txt
Admin
llms.txt is a proposal for websites to include a markdown file providing background information and...
Tokens:6,132
Snippets:51
Trust Score:9.3
Update:1 week ago
Show doc for...
Tokens:
Raw
### Start ed and enable helpful error messages

Source: https://llmstxt.org/ed

Starts the ed editor and enables helpful error messages for the walkthrough.

```sh
$ ed
* H
```

--------------------------------

### Get CLI help with llms_txt2ctx -h

Source: https://llmstxt.org/intro.html

Display help for the llms_txt2ctx command-line tool.

```sh
llms_txt2ctx -h
```

--------------------------------

### Complete llms.txt parser in under 20 lines

Source: https://llmstxt.org/intro.html

A complete parser for llms.txt files with no dependencies, provided as an implementation example. It uses regex and itertools to parse the file into a dict.

```python
from pathlib import Path
import re,itertools

def chunked(it, chunk_sz):
    it = iter(it)
    return iter(lambda: list(itertools.islice(it, chunk_sz)), [])

def parse_llms_txt(txt):
    "Parse llms.txt file contents in `txt` to a `dict`"
    def _p(links):
        link_pat = r'-\s*\[(?P<title>[^\]]+)\]\((?P<url>[^\)]+)\)(?::\s*(?P<desc>.*))?'
        return [re.search(link_pat, l).groupdict()
                for l in re.split(r'\n+', links.strip()) if l.strip()]

    start,*rest = re.split(fr'^##\s*(.*?$)', txt, flags=re.MULTILINE)
    sects = {k: _p(v) for k,v in dict(chunked(rest, 2)).items()}
    pat = r'^#\s*(?P<title>.+?$)\n+(?:^>\s*(?P<summary>.+?$)$)?\n+(?P<info>.*)'
    d = re.search(pat, start.strip(), (re.MULTILINE|re.DOTALL)).groupdict()
    d['sections'] = sects
    return d
```

--------------------------------

### Install llms-txt with pip

Source: https://llmstxt.org/intro.html

Install the llms-txt package to use the CLI and Python API.

```sh
pip install llms-txt
```

--------------------------------

### Save app to file and quit ed

Source: https://llmstxt.org/ed

Saves the generated app to hello_world.py and quits ed.

```sh
*w hello_world.py
5
*q
```

--------------------------------

### Output of print(ctx[:300])

Source: https://llmstxt.org/intro.html

Shows the beginning of the generated XML context.

```text
<project title="FastHTML" summary='FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore&#39;s `FT` "FastTags" into a library for creating server-rendered hypermedia applications.'>
Remember:

- Use `serve()` for running uvicorn (`if __name__ == "__main__"` is not
```

--------------------------------

### Print first 300 characters of context

Source: https://llmstxt.org/intro.html

Print the first 300 characters of the generated XML context.

```python
print(ctx[:300])
```

--------------------------------

### Generate FastHTML app with x command

Source: https://llmstxt.org/ed

Invokes the hypothetical x (eXecute AI) command to generate a FastHTML app based on the prompt and previously loaded context.

```sh
* x Create a simple FastHTML app which outputs 'Hello, World!', in a <div>.
Analyzing context and prompt...
Generating FastHTML app...
App written to buffer.
```

--------------------------------

### Output of parsed.title and parsed.summary

Source: https://llmstxt.org/intro.html

Shows the title and summary values from the parsed file.

```text
('FastHTML',
 'FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\'s `FT` "FastTags" into a library for creating server-rendered hypermedia applications.')
```

--------------------------------

### Output of list(parsed.sections)

Source: https://llmstxt.org/intro.html

Shows the section names.

```text
['Docs', 'Examples', 'Optional']
```

--------------------------------

### Create XML context with create_ctx

Source: https://llmstxt.org/intro.html

Create an LLM context file with XML sections, suitable for systems such as Claude. This is what the CLI calls behind the scenes.

```python
ctx = create_ctx(samp)
```

--------------------------------

### Load llms.txt from fastht.ml/docs with L command

Source: https://llmstxt.org/ed

Invokes the hypothetical L (load) command to retrieve and process the llms.txt file from fastht.ml/docs. ed checks for the file, parses it, fetches relevant URLs, and creates an XML-based context for Claude. If the file didn't exist, it would fall back to scraping the HTML.

```sh
* L fastht.ml/docs
Checking for /llms.txt at fastht.ml/docs...
Found /llms.txt. Parsing...
Fetching URLs from "Docs" section...  Fetching URLs from "Examples" section...
Skipping "Optional" section for brevity.
Creating XML-based context for Claude...  Context created and loaded.
```

--------------------------------

### Access parsed title and summary

Source: https://llmstxt.org/intro.html

Access the title and summary fields from the parsed data.

```python
parsed.title,parsed.summary
```

--------------------------------

### Output of list(parsed)

Source: https://llmstxt.org/intro.html

Shows the keys of the parsed data structure.

```text
['title', 'summary', 'info', 'sections']
```

--------------------------------

### List parsed sections

Source: https://llmstxt.org/intro.html

List the section names from the parsed data.

```python
list(parsed.sections)
```

--------------------------------

### Read sample llms.txt file

Source: https://llmstxt.org/intro.html

Read the contents of an llms.txt file into a string for parsing.

```python
samp = Path('llms-sample.txt').read_text()
```

--------------------------------

### View generated app line count and contents

Source: https://llmstxt.org/ed

After generating the FastHTML app, the user views the line count (n) and contents (p) of the buffer.

```sh
* n
5
* p
from fasthtml.common import *
app,rt = fast_app()
@rt
def index(): return div("Hello, World!")
serve()
```

--------------------------------

### Output of parsed.sections.Optional[0]

Source: https://llmstxt.org/intro.html

Shows the dictionary for the first optional section entry.

```json
{ 'desc': 'A subset of the Starlette documentation useful for FastHTML '
          'development.',
  'title': 'Starlette full documentation',
  'url': 'https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md'}
```

--------------------------------

### Access optional section entry

Source: https://llmstxt.org/intro.html

Access the first entry of the Optional section.

```python
parsed.sections.Optional[0]
```

--------------------------------

### Convert llms.txt to XML context with llms_txt2ctx

Source: https://llmstxt.org/intro.html

Convert an llms.txt file to XML context and save to llms.md. Use --optional True to include the optional section.

```sh
llms_txt2ctx llms.txt > llms.md
```

--------------------------------

### HTTP Link header for markdown and llms.txt

Source: https://llmstxt.org/index

Use this HTTP Link header to point to the markdown version of a page and the llms.txt file that describes it. Works for non-HTML resources and can be added in web server or CDN configuration.

```http
Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"
```

--------------------------------

### Import llms_txt module

Source: https://llmstxt.org/intro.html

Import the llms_txt Python module to use its API.

```python
from llms_txt import *
```

--------------------------------

### Parse llms.txt file with parse_llms_file

Source: https://llmstxt.org/intro.html

Parse an llms.txt file into a data structure with sections. Optionally pass optional=True to include the optional section.

```python
parsed = parse_llms_file(samp)
list(parsed)
```
Tokens:
Raw