Funcparserlib

https://funcparserlib.pirx.ru/
Admin
Funcparserlib is a recursive descent parsing library for Python based on functional combinators,...
Tokens:26,316
Snippets:201
Trust Score:6.3
Update:1 week ago
Show doc for...
Tokens:
Raw
### Run examples via doctest

Source: https://funcparserlib.pirx.ru/getting-started

Command to run the code examples from the guide using Python's doctest module.

```bash
python3 -m doctest -v docs/getting-started/*.md
```

--------------------------------

### Tokenizer setup for examples

Source: https://funcparserlib.pirx.ru/getting-started/tips-and-tricks

Defines a tokenizer for floats, ints, operators, and whitespace, and a helper for operator tokens. Used as the basis for all examples in this chapter.

```python
>>> from typing import List
>>> from funcparserlib.lexer import make_tokenizer, TokenSpec, Token
>>> from funcparserlib.parser import tok, Parser, many, forward_decl, finished


>>> def tokenize(s: str) -> List[Token]:
...     specs = [
...         TokenSpec("whitespace", r"\s+"),
...         TokenSpec("float", r"[+\-]?\d+\.\d*([Ee][+\-]?\d+)*"),
...         TokenSpec("int", r"[+\-]?\d+"),
...         TokenSpec("op", r"(\*\*)|[+\-*/()]"),
...     ]
...     tokenizer = make_tokenizer(specs)
...     return [t for t in tokenizer(s) if t.type != "whitespace"]


>>> def op(name: str) -> Parser[Token, str]:
...     return tok("op", name)
```

--------------------------------

### Install funcparserlib with pip

Source: https://funcparserlib.pirx.ru/

Install funcparserlib from PyPI. The library has no dependencies on other libraries.

```bash
$ pip install funcparserlib

```

--------------------------------

### Tokenize example expression

Source: https://funcparserlib.pirx.ru/getting-started/parsing

Shows the output of the tokenize function on a sample arithmetic expression, producing a list of Token objects.

```python
>>> from pprint import pprint


>>> pprint(tokenize("3.1415926 * (2 + 7.18281828e-1) * 42"))
[Token('float', '3.1415926'),
 Token('op', '*'),
 Token('op', '('),
 Token('int', '2'),
 Token('op', '+'),
 Token('float', '7.18281828e-1'),
 Token('op', ')'),
 Token('op', '*'),
 Token('int', '42')]

```

--------------------------------

### Example expressions to parse

Source: https://funcparserlib.pirx.ru/getting-started

A list of expression strings that the parser should be able to handle, including integers, floats, and operators.

```text
0
1 + 2 + 3
-1 + 2 ** 32
3.1415926 * (2 + 7.18281828e-1) * 42
```

--------------------------------

### make_tokenizer example with TokenSpec and LexerError

Source: https://funcparserlib.pirx.ru/api/lexer

Shows how to create a tokenizer with make_tokenizer and TokenSpec, tokenize a string, filter out spaces, and handle a LexerError when the input cannot be tokenized.

```python
>>> tokenize = make_tokenizer([
...     TokenSpec("space", r"\s+"),
...     TokenSpec("id", r"\w+"),
...     TokenSpec("op", r"[,!]"),
... ])
>>> text = "Hello, World!"
>>> [t for t in tokenize(text) if t.type != "space"]  # noqa
[Token('id', 'Hello'), Token('op', ','), Token('id', 'World'), Token('op', '!')]
>>> text = "Bye?"
>>> list(tokenize(text))
Traceback (most recent call last):
    ...
lexer.LexerError: cannot tokenize data: 1,4: "Bye?"
```

--------------------------------

### One or more repetitions with oneplus

Source: https://funcparserlib.pirx.ru/api/parser

Shows the oneplus combinator that applies a parser one or more times, returning a list of parsed values. The examples demonstrate successful parsing of one or two 'x' tokens, and a failure when the input does not start with 'x'.

```python
>>> expr = oneplus(a("x"))
>>> expr.parse("x")
['x']
>>> expr.parse("xx")
['x', 'x']
>>> expr.parse("y")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'y', expected: 'x'

```

--------------------------------

### Example numeric expressions

Source: https://funcparserlib.pirx.ru/

Example expression strings in the numeric expression language with Python-like syntax.

```text
0
1 + 2 + 3
-1 + 2 ** 32
3.1415926 * (2 + 7.18281828e-1) * 42

```

--------------------------------

### Test the expression parser

Source: https://funcparserlib.pirx.ru/getting-started

A series of doctest examples that parse various expressions and show the resulting parse trees.

```python
>>> parse(tokenize("0"))
0

>>> parse(tokenize("1 + 2 + 3"))
BinaryExpr(op='+', left=BinaryExpr(op='+', left=1, right=2), right=3)

>>> parse(tokenize("-1 + 2 ** 32"))
BinaryExpr(op='+', left=-1, right=BinaryExpr(op='**', left=2, right=32))

>>> parse(tokenize("3.1415926 * (2 + 7.18281828e-1) * 42"))
BinaryExpr(op='*', left=BinaryExpr(op='*', left=3.1415926, right=BinaryExpr(op='+', left=2, right=0.718281828)), right=42)
```

--------------------------------

### Parse output for example expression

Source: https://funcparserlib.pirx.ru/

Output of parse() for the tokenized expression, showing the resulting BinaryExpr tree built by the parser created with funcparserlib.parser.

```python
BinaryExpr(
    op='*',
    left=BinaryExpr(
        op='*',
        left=3.1415926,
        right=BinaryExpr(op='+', left=2, right=0.718281828),
    ),
    right=42,
)

```

--------------------------------

### Tokenize output for example expression

Source: https://funcparserlib.pirx.ru/

Output of tokenize() for the expression 3.1415926 * (2 + 7.18281828e-1) * 42 using the tokenizer created with funcparserlib.lexer.

```python
[
    Token('float', '3.1415926'),
    Token('op', '*'),
    Token('op', '('),
    Token('int', '2'),
    Token('op', '+'),
    Token('float', '7.18281828e-1'),
    Token('op', ')'),
    Token('op', '*'),
    Token('int', '42'),
]

```

--------------------------------

### Choice combination with |

Source: https://funcparserlib.pirx.ru/api/parser

Shows the choice combinator | that tries the first parser and falls back to the second if the first fails. The example demonstrates successful parsing of 'x' and 'y', and a failure with an error message listing both expected tokens when neither matches.

```python
>>> expr = a("x") | a("y")
>>> expr.parse("x")
'x'
>>> expr.parse("y")
'y'
>>> expr.parse("z")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'z', expected: 'x' or 'y'

```

--------------------------------

### Sequential combination with +

Source: https://funcparserlib.pirx.ru/api/parser

Shows how the + combinator sequentially combines parsers, returning a tuple of parsed values. The first two examples demonstrate successful parsing of two and three tokens, while the third shows a failure when the input does not match the expected token.

```python
>>> expr = a("x") + a("y")
>>> expr.parse("xy")
('x', 'y')

```

```python
>>> expr = a("x") + a("y") + a("z")
>>> expr.parse("xyz")
('x', 'y', 'z')

```

```python
>>> expr = a("x") + a("y")
>>> expr.parse("xz")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'z', expected: 'y'

```

--------------------------------

### Transforming results with >>

Source: https://funcparserlib.pirx.ru/api/parser

Illustrates the >> combinator that transforms the parsed result using a function. The example defines a function to lowercase the parsed value and shows that both 'D' and 'd' are converted to 'd'.

```python
>>> def make_canonical_name(s):
...     return s.lower()
>>> expr = (a("D") | a("d")) >> make_canonical_name
>>> expr.parse("D")
'd'
>>> expr.parse("d")
'd'

```

--------------------------------

### Optional parsing with maybe

Source: https://funcparserlib.pirx.ru/api/parser

Shows the maybe combinator that returns None if the inner parser fails. The example demonstrates that parsing 'x' returns 'x', while parsing 'y' returns None.

```python
>>> expr = maybe(a("x"))
>>> expr.parse("x")
'x'
>>> expr.parse("y") is None
True

```

--------------------------------

### Zero or more repetitions with many

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates the many combinator that applies a parser zero or more times, returning a list of parsed values. The examples show parsing one, two, and three 'x' tokens, and an empty list when the input does not start with 'x'.

```python
>>> expr = many(a("x"))
>>> expr.parse("x")
['x']
>>> expr.parse("xx")
['x', 'x']
>>> expr.parse("xxxy")  # noqa
['x', 'x', 'x']
>>> expr.parse("y")
[]

```

--------------------------------

### Define parser for nested parenthesized numbers

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Initial parser definition using forward_decl and the number | (op("(") + p + op(")")) grammar. This is the starting point before applying the -p combinator.

```python
>>> p = forward_decl()
>>> p.define(number | (op("(") + p + op(")")))

```

--------------------------------

### Example parse tree structure

Source: https://funcparserlib.pirx.ru/getting-started

A visual representation of the tree structure that the parser produces for the expression '3.1415926 * (2 + 7.18281828e-1) * 42'.

```text
BinaryExpr('*')
|-- BinaryExpr('*')
|   |-- 3.1415926
|   `-- BinaryExpr('+')
|       |-- 2
|       `-- 0.718281828
`-- 42
```

--------------------------------

### Ignoring parsed values with -p

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates the -p (negation) combinator that ignores the parsed value of a parser in a sequential combination. The examples show how skipped parsers affect the result, including cases where the result is a single value instead of a tuple, and failure cases when the input does not match the expected token.

```python
>>> expr = -a("x") + a("y")
>>> expr.parse("xy")
'y'

```

```python
>>> expr = a("x") + -a("y")
>>> expr.parse("xy")
'x'

```

```python
>>> expr = a("x") + -a("y") + a("z")
>>> expr.parse("xyz")
('x', 'z')

```

```python
>>> expr = -a("x") + a("y") + -a("z")
>>> expr.parse("xyz")
'y'

```

```python
>>> expr = -a("x") + a("y")
>>> expr.parse("yz")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'y', expected: 'x'

```

```python
>>> expr = a("x") + -a("y")
>>> expr.parse("xz")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'z', expected: 'y'

```

--------------------------------

### Tokenize a simple expression

Source: https://funcparserlib.pirx.ru/getting-started/tokenizing

Demonstrates the tokenizer output for a simple expression. The result is a list of Token objects with type and value.

```python
>>> tokenize("42 + 1337")
[Token('int', '42'), Token('op', '+'), Token('int', '1337')]

```

--------------------------------

### Parser.__or__(other)

Source: https://funcparserlib.pirx.ru/api/parser

Tries alternative parsers in order. Returns the result of the first parser that succeeds, or raises an error if all fail.

```APIDOC
## Parser.__or__(other)

### Description
Choice combination of parsers. Tries this parser first; if it fails, tries the other parser.

### Parameters
- **other** (Parser) - An alternative parser to try if this one fails.

### Examples
```python
>>> expr = a("x") | a("y")
>>> expr.parse("x")
'x'
>>> expr.parse("y")
'y'
>>> expr.parse("z")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'z', expected: 'x' or 'y'
```

```

--------------------------------

### Left recursion example: expr = mul | number

Source: https://funcparserlib.pirx.ru/getting-started/tips-and-tricks

Defines a grammar with left recursion: expr is either mul (expr * expr) or number. This leads to infinite recursion when parsing.

```python
>>> expr = forward_decl()
>>> mul = expr + op("*") + expr
>>> expr.define(mul | number)

```

--------------------------------

### Use finished to expect end of input

Source: https://funcparserlib.pirx.ru/getting-started/parsing

Shows how to combine a parser with the finished parser to ensure the entire input is consumed, and demonstrates the resulting behavior on incomplete and complete input.

```python
>>> expr.parse(tokenize("2 ** (3 ** 4"))
('2', [])

```

```python
>>> from funcparserlib.parser import finished
>>> document = expr + finished

```

```python
>>> document.parse(tokenize("2 ** (3 ** 4"))   # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
    ...
NoParseError: got unexpected end of input, expected: ')'

>>> document.parse(tokenize("2 ** (3 ** 4)"))
('2', [('**', ('(', ('3', [('**', '4')]), ')'))], None)

```

--------------------------------

### a() parser

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates the a() parser that matches a token equal to the given value. Parsing a different value raises NoParseError.

```python
>>> expr = a("x")
>>> expr.parse("x")
'x'
>>> expr.parse("y")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'y', expected: 'x'

```

--------------------------------

### Conflicting alternatives: shorter first (unreachable alternative)

Source: https://funcparserlib.pirx.ru/getting-started/tips-and-tricks

Shows that when a shorter alternative is placed first, parsing '1 2 3' with (number + number) | (number + number + number) returns (1, 2), making the longer alternative unreachable.

```python
>>> p = (number + number) | (number + number + number)

>>> p.parse(tokenize("1 2 3"))
(1, 2)

```

--------------------------------

### Using pretty_tree on a list with custom kids and show functions

Source: https://funcparserlib.pirx.ru/api/util

Shows output of pretty_tree for a nested list, with kids returning child lists and show producing "[]" for lists. Output uses Unix tree style with "|--", "`--" and indentation.

```python
>>> print(pretty_tree(
...     ["foo", ["bar", "baz"], "quux"],
...     lambda obj: obj if isinstance(obj, list) else [],
...     lambda obj: "[]" if isinstance(obj, list) else str(obj),
... ))
[]
|-- foo
|-- []
|   |-- bar
|   `-- baz
`-- quux
```

--------------------------------

### TokenSpec.__init__(type, pattern, flags=0)

Source: https://funcparserlib.pirx.ru/api/lexer

Initializes a TokenSpec object, which is a token specification for generating a lexer via make_tokenizer().

```APIDOC
## TokenSpec.__init__(type, pattern, flags=0)

### Description
Initialize a TokenSpec object. A token specification for generating a lexer via make_tokenizer().

### Parameters
- **type** (str) - Required - User-defined type of the token (e.g. "name", "number", "operator").
- **pattern** (str) - Required - Regexp for matching this token type.
- **flags** (int) - Optional - Regexp flags, the second argument of re.compile(). Default: 0.
```

--------------------------------

### Test grammar with sample input

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Parses the expression `1 * (2 + 0) ** 3` and shows the raw parse result, which is a nested tuple/list structure.

```python
>>> document.parse(tokenize("1 * (2 + 0) ** 3"))
(1, [], [('*', (2, [], [], [('+', (0, [], []))], [('**', 3)]))], [])

```

--------------------------------

### Import tokenizer dependencies

Source: https://funcparserlib.pirx.ru/getting-started/tokenizing

Imports the required types and functions for tokenizing with funcparserlib.lexer.

```python
>>> from typing import List
>>> from funcparserlib.lexer import make_tokenizer, TokenSpec, Token

```

--------------------------------

### make_tokenizer(specs)

Source: https://funcparserlib.pirx.ru/api/lexer

Creates a tokenizer function that tokenizes text based on the given regexp specs. The tokenizer takes a string and returns an iterable of Token objects, or raises LexerError if it cannot tokenize the string.

```APIDOC
## make_tokenizer(specs)

### Description
Make a function that tokenizes text based on the regexp specs. The returned tokenizer takes a string and returns an iterable of Token objects, or raises LexerError if it cannot tokenize the string according to its token specs.

### Signature
`make_tokenizer(specs: Sequence[TokenSpec | Tuple]) -> Callable[[str], Iterable[Token]]`

### Parameters
- **specs** (Sequence[TokenSpec | Tuple]) - Required - A sequence of token specifications. Each spec is a TokenSpec instance, or for legacy reasons, a tuple of (type, args) where type sets the Token.type and args are positional arguments for re.compile(): either (pattern,) or (pattern, flags).

### Returns
A tokenizer function that takes a string and returns an iterable of Token objects.

### Raises
- **LexerError** - If the tokenizer cannot tokenize the string according to its token specs.

### Example
```python
>>> tokenize = make_tokenizer([
...     TokenSpec("space", r"\s+"),
...     TokenSpec("id", r"\w+"),
...     TokenSpec("op", r"[,!]"),
... ])
>>> text = "Hello, World!"
>>> [t for t in tokenize(text) if t.type != "space"]  # noqa
[Token('id', 'Hello'), Token('op', ','), Token('id', 'World'), Token('op', '!')]
>>> text = "Bye?"
>>> list(tokenize(text))
Traceback (most recent call last):
    ...
lexer.LexerError: cannot tokenize data: 1,4: "Bye?"
```
```

--------------------------------

### oneplus(p)

Source: https://funcparserlib.pirx.ru/api/parser

Repeatedly applies a parser one or more times. Fails if the parser cannot match at least once.

```APIDOC
## oneplus(p)

### Description
Returns a parser that applies `p` one or more times until it fails. Fails if `p` cannot be applied at least once.

### Parameters
- **p** (Parser) - The parser to repeat.

### Examples
```python
>>> expr = oneplus(a("x"))
>>> expr.parse("x")
['x']
>>> expr.parse("xx")
['x', 'x']
>>> expr.parse("y")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: 'y', expected: 'x'
```

```

--------------------------------

### Parser monad combinators using bind() and pure()

Source: https://funcparserlib.pirx.ru/api/parser

Shows how parser combinators can be expressed in terms of bind() and pure(), though the source notes this would be inefficient in Python. The class definition is incomplete and relies on external definitions of bind() and pure().

```python
# noinspection PyUnresolvedReferences
class Parser:
    def __add__(self, other):
        return self.bind(lambda x: other.bind(lambda y: pure((x, y))))

    def __rshift__(self, other):
        return self.bind(lambda x: pure(x))

```

--------------------------------

### funcparserlib.util.pretty_tree(x, kids, show)

Source: https://funcparserlib.pirx.ru/api/util

Returns a pseudo-graphic tree representation of an object, similar to the Unix tree command. The function takes three arguments: the object to represent, a function to list its children, and a function to produce a textual representation of each object.

```APIDOC
## `funcparserlib.util.pretty_tree(x, kids, show)`

### Description
Return a pseudo-graphic tree representation of the object `x` similar to the `tree` command in Unix.

### Signature
`pretty_tree(x, kids, show)`

Type: `(T, Callable[[T], List[T]], Callable[[T], str]) -> str`

### Parameters
- **x** (T) - The object to represent as a tree.
- **kids** (Callable[[T], List[T]]) - A function that returns the children of the object.
- **show** (Callable[[T], str]) - A function that returns a textual representation of the object.

### Return Value
- **str** - A string containing the tree representation.

### Example
```
>>> print(pretty_tree(
...     ["foo", ["bar", "baz"], "quux"],
...     lambda obj: obj if isinstance(obj, list) else [],
...     lambda obj: "[]" if isinstance(obj, list) else str(obj),
... ))
[]
|-- foo
|-- []
|   |-- bar
|   `-- baz
`-- quux
```
```

--------------------------------

### Complete source code of the expression parser

Source: https://funcparserlib.pirx.ru/getting-started

The full implementation of the numeric expression parser using funcparserlib, including tokenization, grammar definition, and parsing logic.

```python
>>> from typing import List, Tuple, Union
>>> from dataclasses import dataclass
>>> from funcparserlib.lexer import make_tokenizer, TokenSpec, Token
>>> from funcparserlib.parser import tok, Parser, many, forward_decl, finished


>>> @dataclass
... class BinaryExpr:
...     op: str
...     left: "Expr"
...     right: "Expr"


>>> Expr = Union[BinaryExpr, int, float]


>>> def tokenize(s: str) -> List[Token]:
...     specs = [
...         TokenSpec("whitespace", r"\s+"),
...         TokenSpec("float", r"[+\-]?\d+\.\d*([Ee][+\-]?\d+)*"),
...         TokenSpec("int", r"[+\-]?\d+"),
...         TokenSpec("op", r"(\*\*)|[+\-*/()]"),
...     ]
...     tokenizer = make_tokenizer(specs)
...     return [t for t in tokenizer(s) if t.type != "whitespace"]


>>> def parse(tokens: List[Token]) -> Expr:
...     int_num = tok("int") >> int
...     float_num = tok("float") >> float
...     number = int_num | float_num
...
...     expr: Parser[Token, Expr] = forward_decl()
...     parenthesized = -op("(") + expr + -op(")")
...     primary = number | parenthesized
...     power = primary + many(op("**") + primary) >> to_expr
...     term = power + many((op("*") | op("/")) + power) >> to_expr
...     sum = term + many((op("+") | op("-")) + term) >> to_expr
...     expr.define(sum)
...
...     document = expr + -finished
...
...     return document.parse(tokens)


>>> def op(name: str) -> Parser[Token, str]:
...     return tok("op", name)


>>> def to_expr(args: Tuple[Expr, List[Tuple[str, Expr]]]) -> Expr:
...     first, rest = args
...     result = first
...     for op, expr in rest:
...         result = BinaryExpr(op, result, expr)
...     return result
```

--------------------------------

### Define a forward declaration and use it in a parser

Source: https://funcparserlib.pirx.ru/getting-started/parsing

Creates a forward declaration 'p', defines it as a parser that accepts numbers or parenthesized expressions, and then tests it with nested parentheses.

```python
>>> from funcparserlib.parser import forward_decl
>>> p = forward_decl()
>>> p.define(number | (op("(") + p + op(")")))

```

```python
>>> p.parse(tokenize("1"))
'1'

>>> p.parse(tokenize("(1)"))
('(', '1', ')')

>>> p.parse(tokenize("((1))"))
('(', ('(', '1', ')'), ')')

```

--------------------------------

### Define tokenizer with make_tokenizer()

Source: https://funcparserlib.pirx.ru/getting-started/tokenizing

Defines a tokenizer for numeric expressions using make_tokenizer() and TokenSpec. The specs order matters: float before int, and ** before *. Whitespace tokens are filtered out.

```python
>>> def tokenize(s: str) -> List[Token]:
...     specs = [
...         TokenSpec("whitespace", r"\s+"),
...         TokenSpec("float", r"[+\-]?\d+\.\d*([Ee][+\-]?\d+)*"),
...         TokenSpec("int", r"[+\-]?\d+"),
...         TokenSpec("op", r"(\*\*)|[+\-*/()]"),
...     ]
...     tokenizer = make_tokenizer(specs)
...     return [t for t in tokenizer(s) if t.type != "whitespace"]

```

--------------------------------

### Import tok from funcparserlib.parser

Source: https://funcparserlib.pirx.ru/getting-started/parsing

Imports the `tok` function needed to create token parsers.

```python
>>> from funcparserlib.parser import tok

```

--------------------------------

### tok() with type and value

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates matching a specific token by type and value with tok(). The parser returns the token's string value. A token with the same type but different value raises NoParseError.

```python
>>> expr = tok("op", "=")
>>> expr.parse([Token("op", "=")])
'='
>>> expr.parse([Token("op", "+")])
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: '+', expected: '='

```

--------------------------------

### Test initial parser on nested parentheses

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Parsing "((1))" with the initial parser returns a nested tuple including the parentheses tokens.

```python
>>> p.parse(tokenize("((1))"))
('(', ('(', 1, ')'), ')')

```

--------------------------------

### Set parser name with named()

Source: https://funcparserlib.pirx.ru/api/parser

Shows how to use the named() method to set a custom name for a parser, which is then accessible via the name attribute. The output demonstrates that the name is set to 'expr'.

```python
>>> expr = (a("x") + a("y")).named("expr")
>>> expr.name
'expr'

```

--------------------------------

### Conflicting alternatives: longest first (correct order)

Source: https://funcparserlib.pirx.ru/getting-started/tips-and-tricks

Shows that placing the longest alternative first allows parsing both '1 2 3' and '1 2' correctly, returning (1, 2, 3) and (1, 2) respectively.

```python
>>> p = (number + number + number) | (number + number)

>>> p.parse(tokenize("1 2 3"))
(1, 2, 3)

>>> p.parse(tokenize("1 2"))
(1, 2)

```

--------------------------------

### Complete tokenizer and parser for numeric expressions

Source: https://funcparserlib.pirx.ru/

Complete source code for a tokenizer and parser for numeric expressions using funcparserlib. It defines a tokenizer with TokenSpec, a recursive descent parser with combinators like tok, many, forward_decl, and finished, and builds a BinaryExpr tree.

```python
from typing import List, Tuple, Union
from dataclasses import dataclass

from funcparserlib.lexer import make_tokenizer, TokenSpec, Token
from funcparserlib.parser import tok, Parser, many, forward_decl, finished


@dataclass
class BinaryExpr:
    op: str
    left: "Expr"
    right: "Expr"


Expr = Union[BinaryExpr, int, float]


def tokenize(s: str) -> List[Token]:
    specs = [
        TokenSpec("whitespace", r"\s+"),
        TokenSpec("float", r"[+\-]?\d+\.\d*([Ee][+\-]?\d+)*"),
        TokenSpec("int", r"[+\-]?\d+"),
        TokenSpec("op", r"(\*\*)|[+\-*/()]"),
    ]
    tokenizer = make_tokenizer(specs)
    return [t for t in tokenizer(s) if t.type != "whitespace"]


def parse(tokens: List[Token]) -> Expr:
    int_num = tok("int") >> int
    float_num = tok("float") >> float
    number = int_num | float_num

    expr: Parser[Token, Expr] = forward_decl()
    parenthesized = -op("(") + expr + -op(")")
    primary = number | parenthesized
    power = primary + many(op("**") + primary) >> to_expr
    term = power + many((op("*") | op("/")) + power) >> to_expr
    sum = term + many((op("+") | op("-")) + term) >> to_expr
    expr.define(sum)

    document = expr + -finished

    return document.parse(tokens)


def op(name: str) -> Parser[Token, str]:
    return tok("op", name)


def to_expr(args: Tuple[Expr, List[Tuple[str, Expr]]]) -> Expr:
    first, rest = args
    result = first
    for op, expr in rest:
        result = BinaryExpr(op, result, expr)
    return result

```

--------------------------------

### finished

Source: https://funcparserlib.pirx.ru/api/parser

A parser that ensures all input is consumed; throws an error if tokens remain.

```APIDOC
## `finished`

### Description
A parser that throws an exception if there are any unparsed tokens left in the sequence.

### Type
`Parser[Any, None]`

### Examples
```
>>> from funcparserlib.parser import a, finished
>>> expr = a("x") + finished
>>> expr.parse("x")
('x', None)
```

```
>>> expr = a("x") + finished
>>> expr.parse("xy")
Traceback (most recent call last):
    ...
funcparserlib.parser.NoParseError: got unexpected token: 'y', expected: end of input
```
```

--------------------------------

### Parsing a power expression with `number + op("**") + number`

Source: https://funcparserlib.pirx.ru/getting-started/parsing

Builds a parser for a power expression like `2 ** 32` by combining `number`, `op("**")`, and `number` sequentially. The result is a tuple of the three parsed elements.

```python
>>> power = number + op("**") + number

```

```python
>>> power.parse(tokenize("2 ** 32"))
('2', '**', '32')

```

--------------------------------

### Parse an integer with the number parser

Source: https://funcparserlib.pirx.ru/getting-started/parsing

Parses the tokenized input `"42"` with the `number` parser, returning the string `'42'`.

```python
>>> number.parse(tokenize("42"))
'42'

```

--------------------------------

### Test pretty_expr function

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Parses the same expression and pretty-prints the tree, showing the hierarchical structure.

```python
>>> print(pretty_expr(document.parse(tokenize("3.1415926 * (2 + 7.18281828e-1) * 42"))))
BinaryExpr('*')
|-- BinaryExpr('*')
|   |-- 3.1415926
|   `-- BinaryExpr('+')
|       |-- 2
|       `-- 0.718281828
`-- 42

```

--------------------------------

### finished parser

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates the finished parser that ensures no unparsed tokens remain. Parsing a sequence with extra tokens raises NoParseError.

```python
>>> from funcparserlib.parser import a, finished
>>> expr = a("x") + finished
>>> expr.parse("x")
('x', None)

```

--------------------------------

### Omit type hint for `>>`

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Shows that type hints are optional; the `>>` combinator's type is inferrable, so you can omit it for brevity.

```python
>>> int_num = tok("int") >> int

```

--------------------------------

### forward_decl() recursive parser

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates forward_decl() with define() to create a recursive parser. The parser matches 'x' followed by an optional recursive expression and then 'y'. Parsing incomplete input raises NoParseError.

```python
>>> expr = forward_decl()
>>> expr.define(a("x") + maybe(expr) + a("y"))
>>> expr.parse("xxyy")  # noqa
('x', ('x', None, 'y'), 'y')
>>> expr.parse("xxy")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected end of input, expected: 'y'

```

--------------------------------

### Parser combinator structure diagram

Source: https://funcparserlib.pirx.ru/getting-started/parsing

ASCII diagram illustrating the layered structure of parsers in funcparserlib, showing primitive parsers, parser combinators, and means of abstraction.

```text
┌──────────┬──────────────────────┬───────────┐
│          │ Primitive Parsers    │           │
│          ├──────────────────────┘           │
│          │                                  │
│          │ tok(type, value)  forward_decl() │
│          │                                  │
│          │ a(token)  some(pred)  finished   │
│          │                                  │
│          ├──────────────────────┬───────────┤
│          │ Parser Combinators   │           │
│          ├──────────────────────┘           │
│          │                                  │
│  Parser  │ p1 + p2   p1 | p2   p >> f   -p  │
│  objects │                                  │
│          │ many(p)  oneplus(p)  maybe(p)    │
│          │                                  │
│          ├──────────────────────┬───────────┤
│          │ Means of Abstraction │           │
│          ├──────────────────────┘           │
│          │                                  │
│          │ Python assignments: =            │
│          │                                  │
│          │ Python functions: def            │
└──────────┴──────────────────────────────────┘

```

--------------------------------

### Show token positions with str()

Source: https://funcparserlib.pirx.ru/getting-started/tokenizing

Shows the string representation of tokens, which includes their position in the input text. The position is also available via t.start and t.end.

```python
>>> [str(t) for t in tokenize("42 + 1337")]
["1,1-1,2: int '42'", "1,4-1,4: op '+'", "1,6-1,9: int '1337'"]

```

--------------------------------

### Parser.define(p)

Source: https://funcparserlib.pirx.ru/api/parser

Documentation for the Parser.define method that defines a forward-declared parser.

```APIDOC
## Parser.define(p)

### Description
Defines the parser created earlier as a forward declaration. Use in combination with forward_decl() to define recursive parsers.

### Method
define

### Signature
Parser.define(p) -> None

### Parameters
- **p** (Parser[A, B]) - Required - The parser to define.

### Example
```python
p = forward_decl()
p.define(...)
```
```

--------------------------------

### some() with predicate

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates the some() parser with a predicate. The parser is named 'alpha' for error messages. Parsing a token that fails the predicate raises NoParseError.

```python
>>> expr = some(lambda s: s.isalpha()).named('alpha')
>>> expr.parse("x")
'x'
>>> expr.parse("y")
'y'
>>> expr.parse("1")
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: '1', expected: alpha

```

--------------------------------

### Test number parser with int and float

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Parses "42" and "3.14" to verify the number parser returns the correct types.

```python
>>> number.parse(tokenize("42"))
42

>>> number.parse(tokenize("3.14"))
3.14

```

--------------------------------

### Parser.__neg__()

Source: https://funcparserlib.pirx.ru/api/parser

Creates an ignored parser for use in sequential combinations. Useful for discarding syntax like commas or semicolons.

```APIDOC
## Parser.__neg__()

### Description
Returns a parser that parses the same tokens as the original but whose result is ignored by the sequential `+` combinator.

### Examples
```python
>>> expr = -a("x") + a("y")
>>> expr.parse("xy")
'y'
```

```python
>>> expr = a("x") + -a("y")
>>> expr.parse("xy")
'x'
```

```python
>>> expr = a("x") + -a("y") + a("z")
>>> expr.parse("xyz")
('x', 'z')
```

```

--------------------------------

### Parser.__add__(other)

Source: https://funcparserlib.pirx.ru/api/parser

Combines two parsers sequentially, returning a tuple of their results. Use unary minus or skip() to ignore results of individual parsers.

```APIDOC
## Parser.__add__(other)

### Description
Sequential combination of parsers. Runs this parser, then the other parser, and returns a tuple of parsed values (or a single value if some parsers are skipped).

### Parameters
- **other** (Parser) - The parser to run after this one.

### Examples
```python
>>> expr = a("x") + a("y")
>>> expr.parse("xy")
('x', 'y')
```

```python
>>> expr = a("x") + a("y") + a("z")
>>> expr.parse("xyz")
('x', 'y', 'z')
```

### Errors
- **NoParseError** - If the second parser fails on the remaining input.

```

--------------------------------

### Test int_num parser

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Parses the tokenized string "42" and returns the integer 42.

```python
>>> int_num.parse(tokenize("42"))
42

```

--------------------------------

### many(p)

Source: https://funcparserlib.pirx.ru/api/parser

Repeatedly applies a parser as many times as possible, collecting results into a list. Requires zero or more successful applications.

```APIDOC
## many(p)

### Description
Returns a parser that applies `p` zero or more times until it fails. The result is a list of parsed values.

### Parameters
- **p** (Parser) - The parser to repeat.

### Examples
```python
>>> expr = many(a("x"))
>>> expr.parse("x")
['x']
>>> expr.parse("xx")
['x', 'x']
>>> expr.parse("xxxy")
['x', 'x', 'x']
>>> expr.parse("y")
[]
```

```

--------------------------------

### maybe(p)

Source: https://funcparserlib.pirx.ru/api/parser

Makes a parser optional. The resulting parser returns `None` if the underlying parser fails.

```APIDOC
## maybe(p)

### Description
Returns a parser that either parses `p` successfully or returns `None` if `p` fails.

### Parameters
- **p** (Parser) - The parser to optionally apply.

### Examples
```python
>>> expr = maybe(a("x"))
>>> expr.parse("x")
'x'
>>> expr.parse("y") is None
True
```

```

--------------------------------

### Test parser with -p on various inputs

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Parsing "1", "(1)", and "((1))" now returns just the number, demonstrating that parentheses are skipped.

```python
>>> p.parse(tokenize("1"))
1

>>> p.parse(tokenize("(1)"))
1

>>> p.parse(tokenize("((1))"))
1

```

--------------------------------

### tok() with type only

Source: https://funcparserlib.pirx.ru/api/parser

Demonstrates matching any token of a given type with tok(). The parser returns the token's string value. Parsing a token of a different type raises NoParseError.

```python
>>> expr = tok("expr")
>>> expr.parse([Token("expr", "foo")])
'foo'
>>> expr.parse([Token("expr", "bar")])
'bar'
>>> expr.parse([Token("op", "=")])
Traceback (most recent call last):
    ...
parser.NoParseError: got unexpected token: '=', expected: expr

```

--------------------------------

### Test transformed grammar

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Parses `3.1415926 * (2 + 7.18281828e-1) * 42` and shows the resulting `BinaryExpr` tree.

```python
>>> document.parse(tokenize("3.1415926 * (2 + 7.18281828e-1) * 42"))
BinaryExpr(op='*', left=BinaryExpr(op='*', left=3.1415926, right=BinaryExpr(op='+', left=2, right=0.718281828)), right=42)

```

--------------------------------

### Define tokenizer and parser for calculator expressions

Source: https://funcparserlib.pirx.ru/getting-started/parse-tree

Defines the tokenizer and parser for a calculator language supporting floats, ints, and operators. The parser handles precedence with `forward_decl` and `many`.

```python
>>> from typing import List
>>> from funcparserlib.lexer import make_tokenizer, TokenSpec, Token
>>> from funcparserlib.parser import tok, Parser, many, forward_decl, finished


>>> def tokenize(s: str) -> List[Token]:
...     specs = [
...         TokenSpec("whitespace", r"\s+"),
...         TokenSpec("float", r"[+\-]?\d+\.\d*([Ee][+\-]?\d+)*"),
...         TokenSpec("int", r"[+\-]?\d+"),
...         TokenSpec("op", r"(\*\*)|[+\-*/()]"),
...     ]
...     tokenizer = make_tokenizer(specs)
...     return [t for t in tokenizer(s) if t.type != "whitespace"]


>>> def op(name: str) -> Parser[Token, str]:
...     return tok("op", name)


>>> int_str = tok("int")
>>> float_str = tok("float")
>>> number = int_str | float_str
>>> expr = forward_decl()
>>> parenthesized = op("(") + expr + op(")")
>>> primary = number | parenthesized
>>> power = primary + many(op("**") + primary)
>>> expr.define(power)
>>> document = expr + finished

```

--------------------------------

### Parser.parse(tokens)

Source: https://funcparserlib.pirx.ru/api/parser

Documentation for the Parser.parse method that parses a sequence of tokens and returns the parsed value.

```APIDOC
## Parser.parse(tokens)

### Description
Parses a sequence of tokens and returns the parsed value. It raises NoParseError if parsing fails.

### Method
parse

### Signature
Parser.parse(tokens) -> B

### Parameters
- **tokens** (Sequence[A]) - Required - The sequence of tokens to parse.

### Returns
- The parsed value of type B.

### Errors
- NoParseError - raised if parsing fails.

### Example
```python
# Example usage with a tokenizer
from funcparserlib.lexer import make_tokenizer
# ...
result = parser.parse(tokens)
```
```

--------------------------------

### Parser.named(name)

Source: https://funcparserlib.pirx.ru/api/parser

Documentation for the Parser.named method that assigns a name to a parser for debugging.

```APIDOC
## Parser.named(name)

### Description
Specifies a name for the parser for easier debugging. The name is used in debug-level parsing logs and accessible via the Parser.name attribute.

### Method
named

### Signature
Parser.named(name) -> Parser[A, B]

### Parameters
- **name** (str) - Required - The name to assign to the parser.

### Returns
- A new parser with the specified name.

### Example
```python
>>> expr = (a("x") + a("y")).named("expr")
>>> expr.name
'expr'
```
```
Tokens:
Raw