### Start `ed` and enable helpful error messages

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/ed.md

Initiates the `ed` editor and enables verbose error reporting for debugging purposes. This is a setup step for the demonstration.

```sh
$ ed
* H
```

--------------------------------

### Restaurant llms.txt Example

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/domains.md

An example llms.txt file for a restaurant, including its name, description, weekly hours, and links to menus.

```markdown
# Nate the Great's Grill

> Nate the Great's Grill is a popular destination off of Sesame Street that has been serving the community for over 20 years. We offer the best BBQ for a great price.

Here are our weekly hours:

- Monday - Friday: 9am - 9pm
- Saturday: 11am - 9pm
- Sunday: Closed

## Menus

- [Lunch Menu](https://host/lunch.html.md): Our lunch menu served from 11am to 4pm every day.
- [Dinner Menu](https://host/dinner.html.md): Our dinner menu served from 4pm to 9pm every day.

## Optional

- [Dessert Menu](https://host/dessert.md): A subset of the Starlette docs
```

--------------------------------

### FastHTML llms.txt Example

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx.txt

A cut-down example of an llms.txt file for the FastHTML project, showcasing its structure with project title, description, important notes, and links to documentation and examples.

```markdown
# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.

Important notes:

- Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services
- FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte.

## Docs

- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://raw.githubusercontent.com/path/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options

## Examples

- [Todo list application](https://raw.githubusercontent.com/path/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.

```

--------------------------------

### Parse Example Link

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Demonstrates parsing a sample markdown link with title, URL, and description using the `parse_link` function.

```python
parse_link(link)
```

--------------------------------

### Install llms-txt Package

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Install the llms-txt package using pip. This makes the `llms_txt2ctx` CLI tool available.

```shell
pip install llms-txt
```

--------------------------------

### Restaurant Lunch Menu Example

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/domains.md

An example lunch menu for a restaurant, presented in a markdown table format, detailing items and prices for 'By The Pound', 'Drinks', and 'Sides'.

```markdown
## By The Pound

| Item              | Price         |
| --------------    | -----------   |
| Brisket           | 34            |
| Pork Spare Ribs   | 30            |
| Pulled Pork       | 28            |

## Drinks

| Item              | Price         |
| --------------    | -----------   |
| Iced Tea          | 3             |
| Mexican Coke      | 3             |

## Sides

| Item              | Price         |
| --------------    | -----------   |
| Potato Salad      | 4             |
| Slaw              | 4             |
```

--------------------------------

### FastHTML LLMs.txt Example

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

A cut-down version of the llms.txt file for the FastHTML project. It includes the project title, description, important notes, and links to documentation and examples.

```markdown
# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.

Important notes:

- Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services
- FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte.

## Docs

- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://raw.githubusercontent.com/path/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options

## Examples

- [Todo list application](https://raw.githubusercontent.com/path/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.
```

--------------------------------

### Get Help for llms_txt2ctx CLI

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Use the -h flag to display help information for the `llms_txt2ctx` command-line tool.

```shell
llms_txt2ctx -h
```

--------------------------------

### Test llms.txt Parsing in JavaScript

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llmstxt-js.html

Example of how to use the parseLLMsTxt function with sample input and display the resulting JSON output. Ensure the DOM elements with IDs 'input' and 'output' exist.

```javascript
// Test input const testInput = 
# Title > Optional description goes here Optional details go here ## Section name - [Link title](https://link_url): Optional link details ## Optional - [Link title](https://link_url)`;
// Display input document.getElementById('input').textContent = testInput;
// Parse and display output const result = parseLLMsTxt(testInput);
document.getElementById('output').textContent = JSON.stringify(result, null, 2);
```

--------------------------------

### llms.txt Markdown Structure Example

Source: https://github.com/answerdotai/llms-txt/blob/main/README.md

This example demonstrates the basic structure of an llms.txt file, including the required H1 title, optional blockquote, general markdown sections, and file lists delimited by H2 headers.

```markdown
# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)
```

--------------------------------

### List Parsed Sections

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Get a list of the main section titles (e.g., 'Docs', 'Examples') from the parsed llms.txt data.

```python
list(parsed.sections)
```

--------------------------------

### Call Get Sizes Function

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Invokes the `get_sizes` function with a context object and prints the resulting dictionary of section sizes.

```python
get_sizes(ctx)
```

--------------------------------

### Split text by markdown headers

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Splits a string into a starting section and a list of subsequent sections based on markdown headers (##). Useful for parsing structured text files.

```python
start,*rest = re.split(fr'^##\s*(.*?$)', sections, flags=re.MULTILINE)
start
```

```python
rest
```

--------------------------------

### Get LLM Context Section Sizes

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Calculates and returns the size of each section within an LLM context. It iterates through context children, identifying those with tags and extracting the length of their first child's content.

```python
#| export
def get_sizes(ctx):
    """Get the size of each section of the LLM context"""
    return {o.tag:{p.title:len(p.children[0]) for p in o.children} for o in ctx.children if hasattr(o,'tag')}
```

--------------------------------

### Parse LLM text into structured sections

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Parses a larger text block, likely containing LLM-related information, into a starting section and a dictionary of structured sections. It utilizes `_parse_links` for processing individual link sections.

```python
#| export
def _parse_llms(txt):
    start,*rest = re.split(fr'^##\s*(.*?$)', txt, flags=re.MULTILINE)
    d = dict(chunked(rest, 2))
    sects = {k: _parse_links(v) for k,v in d.items()}
    return start.strip(),sects
```

```python
start, sects = _parse_llms(samp)
start
```

--------------------------------

### Create Project Context

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Parses LLM text, creates a project context, and returns it as an XML string. Optionally skips the 'optional' section and uses multiple workers for processing.

```python
skip = '' if optional else 'Optional'
    sections = [_section(k, v, n_workers=n_workers) for k,v in d.sections.items() if k!=skip]
    return Project(title=d.title, summary=d.summary)(d.info, *sections)
```

--------------------------------

### Fetch documentation content from local or remote URL

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Fetches the content of a documentation file. It prioritizes fetching from a local file if the project is part of an nbdev repository and the file exists locally. Otherwise, it fetches the content from the provided URL using httpx.

```python
#| export
def get_doc_content(url):
    """Fetch content from local file if in nbdev repo."""
    if (path:=_get_config()):
        relative_path = urlparse(url).path.lstrip('/')
        local_path = _local_docs_pth(path) / relative_path
        if local_path.exists(): return local_path.read_text()
    return httpx.get(url).text
```

--------------------------------

### Command-Line Interface for LLM Text to Context

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

A command-line tool that reads a file, converts its content to a project context XML, and prints it. It supports options to include/exclude optional sections and specify the number of workers. Output can be saved to a file.

```python
#| export
@call_parse
def llms_txt2ctx(
    fname:str, # File name to read
    optional:bool_arg=False, # Include 'optional' section?
    n_workers:int=None, # Number of threads to use for parallel downloading
    save_nbdev_fname:str=None #save output to nbdev `{docs_path}` instead of emitting to stdout
):
    """Print a `Project` with a `Section` for each H2 part in file read from `fname`, optionally skipping the 'optional' section."""
    ctx = create_ctx(Path(fname).read_text(), optional=optional, n_workers=n_workers)
    if save_nbdev_fname and (cfg:=_get_config()):
        (_local_docs_pth(cfg) / save_nbdev_fname).mk_write(ctx)
    else: print(ctx)
```

--------------------------------

### Create a Project with sections from dictionary

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Initializes a `Project` object, creating a `Section` for each H2-style entry in the input dictionary. It can optionally skip a specified 'optional' section.

```python
#| export
def mk_ctx(d, optional=True, n_workers=None):
    """Create a `Project` with a `Section` for each H2 part in `d`, optionally skipping the 'optional' section."""
```

--------------------------------

### Import Project Dependencies

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Imports necessary utilities from fastcore, httpx, and urllib for project functionality.

```python
#| export
from fastcore.utils import *
from fastcore.xml import *
from fastcore.script import *
import httpx
from urllib.parse import urlparse
```

--------------------------------

### Import necessary XML and documentation utilities

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Imports core classes and functions from the `fastcore.xml` module, including `Sections`, `Project`, and `Doc`, which are essential for generating XML-based documentation.

```python
#| export
from fastcore.xml import Sections,Project,Doc
```

--------------------------------

### Load and process `llms.txt` from a URL in `ed`

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/ed.md

Uses the hypothetical `L` command in `ed` to fetch and parse `llms.txt` from a given URL. It processes documentation sections and prepares context for an LLM.

```sh
* L fastht.ml/docs
Checking for /llms.txt at fastht.ml/docs...
Found /llms.txt. Parsing...
Fetching URLs from "Docs" section...  Fetching URLs from "Examples" section...
Skipping "Optional" section for brevity.
Creating XML-based context for Claude...  Context created and loaded.
```

--------------------------------

### Print Partial LLM Context XML

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Print the first 300 characters of the generated LLM context XML to inspect its structure.

```python
print(ctx[:300])
```

--------------------------------

### Create LLM Context File

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

Use `create_ctx` to generate an LLM context file in XML format from the parsed `llms.txt` content. This is used internally by the CLI.

```python
ctx = create_ctx(samp)

```

--------------------------------

### Convert llms.txt to XML Context using CLI

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

Convert an `llms.txt` file to an XML context file and redirect the output to `llms.md`. Use `--optional True` to include the optional section.

```sh
llms_txt2ctx llms.txt > llms.md
```

```sh
llms_txt2ctx --optional True llms.txt > llms.md
```

--------------------------------

### Read Sample llms.txt File

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Reads the content of the 'llms-sample.txt' file and prints the first 480 characters.

```python
samp = Path('llms-sample.txt').read_text()
print(samp[:480])
```

--------------------------------

### Generate FastHTML app using AI in `ed`

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/ed.md

Employs the hypothetical `x` (eXecute AI) command in `ed` to generate a FastHTML web application based on loaded context and a user prompt. The generated code is placed in the editor's buffer.

```sh
* x Create a simple FastHTML app which outputs 'Hello, World!', in a <div>.
Analyzing context and prompt...
Generating FastHTML app...
App written to buffer.
```

--------------------------------

### Create LLM Context XML from llms.txt Content

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Use the `create_ctx` function to generate an LLM context file in XML format from the raw 'llms.txt' content.

```python
ctx = create_ctx(samp)
```

--------------------------------

### Search for Full Markdown Link

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Searches a string for a complete markdown link, including title, URL, and optional description.

```python
search(pat, link)
```

--------------------------------

### Execute LLM Text to Context Command

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Executes the `llms_txt2ctx` command-line tool to process `llms-sample.txt` and redirects the output to `../fasthtml.md`.

```bash
!llms_txt2ctx llms-sample.txt > ../fasthtml.md
```

--------------------------------

### Save and quit `ed`

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/ed.md

Saves the current buffer content to a file named `hello_world.py` and then exits the `ed` editor.

```sh
*w hello_world.py
5
*q
```

--------------------------------

### Create Context from Text

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

A factory function to create a project context from raw text. It parses the text, makes the context, and returns the XML representation. Supports optional sections and parallel processing.

```python
#| export
def create_ctx(txt, optional=False, n_workers=None):
    """A `Project` with a `Section` for each H2 part in `txt`, optionally skipping the 'optional' section."""
    d = parse_llms_file(txt)
    ctx = mk_ctx(d, optional=optional, n_workers=n_workers)
    return to_xml(ctx, do_escape=False)
```

--------------------------------

### Determine local documentation path and configuration

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Helper functions to determine the local path for processed documentation files and to find the project's configuration file (pyproject.toml) by searching parent directories.

```python
#| export
def _local_docs_pth(path): return path/'_proc'
def _get_config(): return find_file_parents('pyproject.toml')
```

--------------------------------

### Print Partial LLM Context

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

Print the first 300 characters of the generated LLM context file. Note the XML escaping for special characters like apostrophes.

```python
print(ctx[:300])

```

--------------------------------

### Convert llms.txt to XML Context using CLI

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Convert an 'llms.txt' file to an XML context file and save the output to 'llms.md'. By default, optional sections are not included.

```shell
llms_txt2ctx llms.txt > llms.md
```

--------------------------------

### Convert Context to XML

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Converts a project context object to an XML string. `do_escape=False` prevents XML escaping.

```python
ctx = mk_ctx(llmsd)
print(to_xml(ctx, do_escape=False)[:260]+'...')
```

--------------------------------

### Convert llms.txt to XML Context with Optional Sections

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Convert an 'llms.txt' file to an XML context file, including the 'optional' section by passing `--optional True`.

```shell
llms_txt2ctx --optional True llms.txt > llms.md
```

--------------------------------

### Create a Doc FT object from URL content

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Creates a `Doc` FastTag (FT) object by retrieving text content from a given URL. It cleans the retrieved text by removing HTML comments and base64 encoded images before creating the `Doc` object.

```python
#| export
def _doc(kw):
    """Create a `Doc` FT object with the text retrieved from `url` as the child, and `kw` as attrs."""
    print(dict(kw))
    url = kw.pop('url')
    txt = get_doc_content(url)
    re_comment = re.compile('^<!--.*-->$', flags=re.MULTILINE)
    re_base64_img = re.compile(r'<img[^>]*src="data:image/[^"].*"[^>]*>')
    txt = '\n'.join([o for o in txt.splitlines() if not re_comment.search(o) and not re_base64_img.search(o)])
    return Doc(txt, **kw)
```

--------------------------------

### Test llms.txt with Anthropic's Claude

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/domains.md

This script tests your llms.txt file against Anthropic's Claude. It requires Python 3.8+ and the 'claudette', 'llms-txt', and 'requests' libraries. The script fetches the llms.txt content and uses it as context for a chat interaction.

```python
# /// script
# requires-python = ">=3.8"
# dependencies = [
#     "claudette",
#     "llms-txt",
#     "requests",
# ]
# ///
from claudette import *
from llms_txt import create_ctx

import requests

model = models[1] # Sonnet 3.5
chat = Chat(model, sp="""You are a helpful and concise assistant."""

url = 'your_url/llms.txt'
text = requests.get(url).text
lLM_ctx = create_ctx(text)
chat(llm_ctx + '\n\nThe above is necessary context for the conversation.')

while True:
    msg = input('Your question about the site: ')
    res = chat(msg)
    print('From Claude:', contents(res))
```

--------------------------------

### Parse Link Without Description

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Demonstrates parsing a markdown link that only includes a title and URL, with no description.

```python
parse_link('-[foo](http://foo)')
```

--------------------------------

### View generated FastHTML app details in `ed`

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/ed.md

Displays the line count and content of the generated FastHTML application within the `ed` editor using the `n` and `p` commands.

```sh
* n
5
* p
from fasthtml.common import *
app,rt = fast_app()
@rt
def index(): return div("Hello, World!")
serve()
```

--------------------------------

### Read llms.txt File Content

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Read the content of an 'llms-sample.txt' file into a string using `pathlib`. This content can then be parsed.

```python
samp = Path('llms-sample.txt').read_text()
```

--------------------------------

### Import Regular Expression Module

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Imports the regular expression module for pattern matching.

```python
#| export
import re
```

--------------------------------

### Write Context to XML File

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Writes the XML representation of a context object to a file named `fasthtml.md`. `to_xml` is used with `do_escape=False`.

```python
Path('../fasthtml.md').write_text(to_xml(ctx, do_escape=False))
```

--------------------------------

### Create a section with parallel Doc objects

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Generates a section containing multiple `Doc` objects, processed in parallel. It takes a section name and a list of items (each expected to have a 'url' and other attributes), and applies the `_doc` function to each item concurrently.

```python
#| export
def _section(nm, items, n_workers=None):
    """Create a section containing a `Doc` object for each child."""
    return ft(nm, *parallel(_doc, items, n_workers=n_workers, threadpool=True))
```

--------------------------------

### Define Link URL Pattern

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Extends the markdown link pattern to capture the URL.

```python
url = named_re('url', r'[^\)]+')
pat += fr'\({url}\)'
```

--------------------------------

### Define Link Description Pattern

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Further extends the markdown link pattern to optionally capture a description following the URL.

```python
desc = named_re('desc', r'.*')
pat += opt_re(fr':\s*{desc}')
```

--------------------------------

### Access Parsed Title and Summary

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Access the 'title' and 'summary' attributes from the parsed llms.txt data structure.

```python
parsed.title,parsed.summary
```

--------------------------------

### Search for Link Title

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Searches the sample text for a markdown link title using the defined pattern.

```python
search(pat, samp)
```

--------------------------------

### Access Specific Optional Section

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Access the content of a specific section, such as 'Optional', from the parsed data. This returns a dictionary with title, url, and description.

```python
parsed.sections.Optional[0]
```

--------------------------------

### Custom llms.txt Parser Implementation

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

A self-contained Python function to parse 'llms.txt' files without external dependencies, demonstrating core parsing logic.

```python
from pathlib import Path
import re,itertools

def chunked(it, chunk_sz):
    it = iter(it)
    return iter(lambda: list(itertools.islice(it, chunk_sz)), [])

def parse_llms_txt(txt):
    """Parse llms.txt file contents in `txt` to a `dict`"""
    def _p(links):
        link_pat = '-\s*\[(?P<title>[^\]]+)\]\((?P<url>[^\)]+)\)(?::\s*(?P<desc>.*))?'
        return [re.search(link_pat, l).groupdict()
                for l in re.split(r'\n+', links.strip()) if l.strip()]

    start,*rest = re.split(fr'^##\s*(.*?$)', txt, flags=re.MULTILINE)
    sects = {k: _p(v) for k,v in dict(chunked(rest, 2)).items()}
    pat = '^#\s*(?P<title>.+?$)\n+(?:^>\s*(?P<summary>.+?$)$)?\n+(?P<info>.*)'
    d = re.search(pat, start.strip(), (re.MULTILINE|re.DOTALL)).groupdict()
    d['sections'] = sects
    return d

```

--------------------------------

### Include Optional Section in CLI Conversion

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx.txt

Convert an 'llms.txt' file to XML context, including the 'optional' section by passing the --optional True flag. This flag ensures that content marked as optional in the input file is processed.

```sh
llms_txt2ctx llms.txt --optional True > llms.md
```

--------------------------------

### Import llms_txt Module

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Import all necessary functions from the llms_txt library for use in Python scripts.

```python
from llms_txt import *

```

--------------------------------

### Python LLMs.txt Parser

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

A concise Python parser for llms.txt files with no external dependencies. It handles sections, links, titles, and summaries.

```python
from pathlib import Path
import re,itertools

def chunked(it, chunk_sz):
    it = iter(it)
    return iter(lambda: list(itertools.islice(it, chunk_sz)), [])

def parse_llms_txt(txt):
    """Parse llms.txt file contents in `txt` to a `dict`"""
    def _p(links):
        link_pat = '-\\s*\[(?P<title>[^\]]+)\]\((?P<url>[^\)]+)\)(?::\s*(?P<desc>.*))?'
        return [re.search(link_pat, l).groupdict()
                for l in re.split(r'\n+', links.strip()) if l.strip()]

    start,*rest = re.split(fr'^##\s*(.*?$)', txt, flags=re.MULTILINE)
    sects = {k: _p(v) for k,v in dict(chunked(rest, 2)).items()}
    pat = '^#\s*(?P<title>.+?$)\n+(?:^>\s*(?P<summary>.+?$)$)?\n+(?P<info>.*)'
    d = re.search(pat, start.strip(), (re.MULTILINE|re.DOTALL)).groupdict()
    d['sections'] = sects
    return d
```

--------------------------------

### Read llms.txt File Content

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

Read the content of an `llms-sample.txt` file into a string using the `Path` object from the `pathlib` module.

```python
samp = Path('llms-sample.txt').read_text()

```

--------------------------------

### Parse Markdown Link Function

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Parses a markdown link string into its components: title, URL, and optional description. Uses regular expressions to extract these parts.

```python
#| export
def parse_link(txt):
    """Parse a link section from llms.txt"""
    title = named_re('title', r'[^\\]+')
    url = named_re('url', r'[^\)]+')
    desc = named_re('desc', r'.*')
    desc_pat = opt_re(fr':\s*{desc}')
    pat = fr'-\s*\[{title}\]\({url}\){desc_pat}'
    return re.search(pat, txt).groupdict()
```

--------------------------------

### Parse LLM file content into an AttrDict

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

A comprehensive function to parse the content of an LLM file. It extracts the main title, summary, and other information from the header, and also parses associated sections, returning the result as a dictionary-like object.

```python
#| export
def parse_llms_file(txt):
    """Parse llms.txt file contents in `txt` to an `AttrDict`"""
    start,sects = _parse_llms(txt)
    title = named_re('title', r'.+?$')
    summ = named_re('summary', '.+?$')
    summ_pat = opt_re(fr'^>\s*{summ}$')
    info = named_re('info', '.*')
    pat = fr'^#\s*{title}\n+{summ_pat}\n+{info}'
    d = search(pat, start, (re.MULTILINE|re.DOTALL))
    d['sections'] = sects
    return dict2obj(d)
```

```python
llmsd = parse_llms_file(samp)
llmsd.summary
```

```python
llmsd.sections.Examples
```

--------------------------------

### Export Core Module

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Exports the core module for the llms_txt library.

```python
#| default_exp core
```

--------------------------------

### Search Text with Regex

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Searches for a regular expression pattern `pat` within text `txt` and returns a dictionary of matched groups. Returns None if no match is found.

```python
def search(pat, txt, flags=0):
    """Dictionary of matched groups in `pat` within `txt`"""
    res = re.search(pat, txt, flags=flags)
    return res.groupdict() if res else None
```

--------------------------------

### Optional Pattern Helper

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Creates a regular expression pattern to optionally match a given string `s`. Useful for making parts of a regex optional.

```python
#| export
def opt_re(s):
    """Pattern to optionally match `s`"""
    return f'(?:{s})?'
```

--------------------------------

### Search text using a combined regex pattern

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Applies a comprehensive regular expression pattern to search within a text. The pattern is designed to capture a title, an optional summary, and the remaining information from the text, supporting multiline and dotall modes.

```python
pat = fr'^#\s*{title}\n+{summ_pat}\n+{info}'
search(pat, start, (re.MULTILINE|re.DOTALL))
```

--------------------------------

### Chunk list into key-value pairs

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Converts a list into a dictionary by pairing consecutive elements. Assumes the list has an even number of elements, where each pair represents a key and its corresponding value.

```python
d = dict(chunked(rest, 2))
d
```

--------------------------------

### Define Link Title Pattern

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Defines a regular expression pattern to capture the title within a markdown link.

```python
title = named_re('title', r'[^\\]+')
pat =  fr'-\s*\[{title}\]'
```

--------------------------------

### Access Parsed llms.txt Data

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

Access the title and summary from the parsed `llms.txt` data structure. Also, list the available sections.

```python
parsed.title,parsed.summary

```

```python
list(parsed.sections)

```

--------------------------------

### Access Parsed llms.txt Metadata

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx.txt

Access the title and summary attributes from the parsed llms.txt data structure. These represent the main title and a brief description of the content.

```python
parsed.title,parsed.summary

```

--------------------------------

### Define regex patterns for document parsing

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Defines regular expression patterns for extracting title, summary, and additional info from a document's header. Includes optional handling for a summary line prefixed with '>'.

```python
title = named_re('title', r'.+?$')
summ = named_re('summary', '.+?$')
summ_pat = opt_re(fr'^>\s*{summ}$')
info = named_re('info', '.*')
```

--------------------------------

### Named Capture Group Helper

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Creates a regular expression pattern to match `pat` within a named capture group `nm`. This simplifies extracting specific parts of a match.

```python
def named_re(nm, pat):
    """Pattern to match `pat` in a named capture group"""
    return f'(?P<{nm}>{pat})'
```

--------------------------------

### Parse llms.txt File Content

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

Use `parse_llms_file` to create a data structure from the `llms.txt` content. Set `optional=True` to include the optional section.

```python
parsed = parse_llms_file(samp)
list(parsed)

```

--------------------------------

### List Sections of Parsed llms.txt

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx.txt

Retrieve a list of section names from the parsed llms.txt data structure. This shows the top-level categories within the file.

```python
list(parsed.sections)

```

--------------------------------

### Hide Development Imports

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Hides development-specific imports for nbdev.

```python
#| hide
from nbdev.showdoc import *
import nbdev; nbdev.nbdev_export()
```

--------------------------------

### Python Parser for LLMs.txt

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx.txt

Parses the content of an llms.txt file into a structured dictionary. Requires Python 3.6+ and standard libraries like 're', 'itertools', and 'pathlib'.

```python
from pathlib import Path
import re,itertools

def chunked(it, chunk_sz):
    it = iter(it)
    return iter(lambda: list(itertools.islice(it, chunk_sz)), [])

def parse_llms_txt(txt):
    """Parse llms.txt file contents in `txt` to a `dict`"""
    def _p(links):
        link_pat = '-\\s*\\[(?P<title>[^\\]+)\\]\((?P<url>[^\\)]+)\\)(?::\\s*(?P<desc>.*))?'
        return [re.search(link_pat, l).groupdict()
                for l in re.split(r'\\n+', links.strip()) if l.strip()]

    start,*rest = re.split(fr'^##\\s*(.*?$)', txt, flags=re.MULTILINE)
    sects = {k: _p(v) for k,v in dict(chunked(rest, 2)).items()}
    pat = '^#\\s*(?P<title>.+?)\\n+(?:^>\\s*(?P<summary>.+?))$)?\\n+(?P<info>.*)'
    d = re.search(pat, start.strip(), (re.MULTILINE|re.DOTALL)).groupdict()
    d['sections'] = sects
    return d
```

--------------------------------

### Parse llms.txt Content in JavaScript

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llmstxt-js.html

Use this function to parse llms.txt formatted strings into a structured JavaScript object. It handles main titles, optional descriptions, and sections with links.

```javascript
function parseLLMsTxt(txt) { function parseLinks(links) { const linkPat = /-\s*\\\[(?<title>[^\\]+)\\\]\((?<url>[^\\)]+)\\)(?::\\s*(?<desc>.*?))?$/gm; return Array.from(links.matchAll(linkPat)).map(match => match.groups); } const [start, ...rest] = txt.split(/^##\s*(.*?)$/m); const sections = Object.fromEntries( Array.from({ length: Math.floor(rest.length / 2) }, (_, i) => [ rest[i * 2], parseLinks(rest[i * 2 + 1]) ]) ); const pat = /^#\s*(?<title>.+?$\n+(?:^>\s*(?<summary>.+?$))?\n+(?<info>.*)/ms; const match = start.trim().match(pat); const result = { ...match.groups, sections }; return result; }
```

--------------------------------

### Access Optional Section Data

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/llms-ctx-full.txt

Access the first item within the 'Optional' section of the parsed `llms.txt` data structure.

```python
parsed.sections.Optional[0]

```

--------------------------------

### Extract and clean links from text

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Extracts and cleans up markdown-formatted links from a given text block. It splits the text by newlines, strips whitespace, and filters out empty lines before parsing each link.

```python
links = d['S1']
links.strip()
```

```python
#| export
def _parse_links(links):
    return [parse_link(l) for l in re.split(r'\n+', links.strip()) if l.strip()]
```

```python
_parse_links(links)
```

--------------------------------

### nbdev Export

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/01_core.ipynb

Hides the `nbdev_export` function call during evaluation but makes it available for export. This is typically used in notebooks to manage code export.

```python
#| hide
#| eval: false
from nbdev import nbdev_export
nbdev_export()
```

--------------------------------

### Parse llms.txt File Content

Source: https://github.com/answerdotai/llms-txt/blob/main/nbs/00_intro.ipynb

Use `parse_llms_file` to parse the content of an llms.txt file into a structured data object. Set `optional=True` to include optional sections.

```python
parsed = parse_llms_file(samp)
list(parsed)
```