### PLY Lexer Initialization and Tokenization Example

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Initializes a PLY lexer and processes input data. This example demonstrates how to create, input data to, and iterate through tokens generated by the lexer.

```python
lexer = lex.lex()
data = "{}"

lexer.input(data)
while True:
    tok = lexer.token()
    if not tok:
        break
    print(tok)
```

--------------------------------

### Lexer Start Conditions (States)

Source: https://context7.com/dabeaz/ply/llms.txt

Demonstrates how to declare and use lexer states for context-sensitive tokenizing, allowing different token rules to be active depending on the current state.

```APIDOC
## Lexer Start Conditions (States) — Context-Sensitive Tokenizing

Declares multiple lexer states so different token rules activate depending on context, mirroring GNU flex start conditions.

```python
import ply.lex as lex

tokens = ('TEXT', 'COMMENT')

# Declare states: 'comment' is exclusive (only its rules apply while active)
states = (
    ('comment', 'exclusive'),
)

# INITIAL state: match the start of a block comment
def t_begin_comment(t):
    r'/'
    t.lexer.comment_start = t.lexer.lexpos
    t.lexer.begin('comment')          # switch to 'comment' state

# 'comment' state: accumulate until closing */
def t_comment_end(t):
    r'*/'
    t.type  = 'COMMENT'
    t.value = t.lexer.lexdata[t.lexer.comment_start - 2 : t.lexer.lexpos]
    t.lexer.begin('INITIAL')          # return to normal state
    return t

def t_comment_body(t):
    r'(.|\n)+'                         # consume everything else
    pass

# Normal text token
def t_TEXT(t):
    r'[^/]+'
    return t

t_comment_ignore = ''
t_ignore = ''

def t_error(t):      t.lexer.skip(1)
def t_comment_error(t): t.lexer.skip(1)

lexer = lex.lex()
lexer.input("hello /* this is a comment */ world")
for tok in lexer:
    print(tok)
# Output:
# LexToken(TEXT,'hello ',1,0)
# LexToken(COMMENT,'/* this is a comment */',1,6)
# LexToken(TEXT,' world',1,29)
```
```

--------------------------------

### g.set_start

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Sets the starting rule for the grammar. Must be called after all productions have been added.

```APIDOC
## g.set_start(start=None)

### Description
Sets the starting rule for the grammar. `start` is a string specifying the name of the start rule. If `start` is omitted, the first grammar rule added with `add_production()` is taken to be the starting rule. This method must always be called after all productions have been added.

### Parameters
* **start** (string, optional) - The name of the start rule. If omitted, the first production added is used.
```

--------------------------------

### Lexer State Example: C Code Parsing

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Example demonstrating the use of lexer states to parse nested C-style code blocks enclosed in curly braces.

```python
import ply.lex as lex

# Declare the states
states = (
  ('ccode','exclusive'),
)

# Match the first '{' Enter ccode state.
def t_ccode(t):
    r'\{'
    t.lexer.code_start = t.lexer.lexpos        # Record the starting position
    t.lexer.level = 1                          # Initial brace level
    t.lexer.begin('ccode')                     # Enter 'ccode' state

# Rules for the 'ccode' state
def t_ccode_lbrace(t):
    r'\{'
    t.lexer.level += 1

def t_ccode_rbrace(t):
    r'\}'
    t.lexer.level -= 1

    # If closing brace, return the code fragment
    if t.lexer.level == 0:
        
```

--------------------------------

### LR Parsing Step-by-Step Example

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

This table illustrates the LR parsing process for the expression '3 + 5 * (10 - 20)'. It shows the stack, input tokens, and actions taken at each step.

```text
Step Symbol Stack           Input Tokens            Action
    ---- ---------------------  ---------------------   -------------------------------
    1                           3 + 5 * ( 10 - 20 )$    Shift 3
    2    3                        + 5 * ( 10 - 20 )$    Reduce factor : NUMBER
    3    factor                   + 5 * ( 10 - 20 )$    Reduce term   : factor
    4    term                     + 5 * ( 10 - 20 )$    Reduce expr : term
    5    expr                     + 5 * ( 10 - 20 )$    Shift +
    6    expr +                     5 * ( 10 - 20 )$    Shift 5
    7    expr + 5                     * ( 10 - 20 )$    Reduce factor : NUMBER
    8    expr + factor                * ( 10 - 20 )$    Reduce term   : factor
    9    expr + term                  * ( 10 - 20 )$    Shift *
    10   expr + term *                  ( 10 - 20 )$    Shift (
    11   expr + term * (                  10 - 20 )$    Shift 10
    12   expr + term * ( 10                  - 20 )$    Reduce factor : NUMBER
    13   expr + term * ( factor              - 20 )$    Reduce term : factor
    14   expr + term * ( term                - 20 )$    Reduce expr : term
    15   expr + term * ( expr                - 20 )$    Shift -
    16   expr + term * ( expr -                20 )$    Shift 20
    17   expr + term * ( expr - 20                )$    Reduce factor : NUMBER
    18   expr + term * ( expr - factor            )$    Reduce term : factor
    19   expr + term * ( expr - term              )$    Reduce expr : expr - term
    20   expr + term * ( expr                     )$    Shift )
    21   expr + term * ( expr )                    $    Reduce factor : (expr)
    22   expr + term * factor                      $    Reduce term : term * factor
```

--------------------------------

### Example p_rule Function and Grammar Entry

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Illustrates a typical p_rule function definition and its corresponding entry in the p.grammar attribute, showing how grammar rules and location information are stored.

```python
def p_expr(p):
    '''expr : expr PLUS expr
            | expr MINUS expr
            | expr TIMES expr
            | expr DIVIDE expr'''
```

```python
('p_expr', [ ('calc.py',10,'expr', ['expr','PLUS','expr']),
                     ('calc.py',11,'expr', ['expr','MINUS','expr']),
                     ('calc.py',12,'expr', ['expr','TIMES','expr']),
                     ('calc.py',13,'expr', ['expr','DIVIDE','expr'])
                   ])
```

--------------------------------

### Parser.out File Structure Example

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

The 'parser.out' file contains a detailed breakdown of the grammar, terminal and nonterminal usage, parsing method, and state machine information. This is useful for analyzing parsing conflicts.

```text
Unused terminals:


    Grammar

    Rule 1     expression -> expression PLUS expression
    Rule 2     expression -> expression MINUS expression
    Rule 3     expression -> expression TIMES expression
    Rule 4     expression -> expression DIVIDE expression
    Rule 5     expression -> NUMBER
    Rule 6     expression -> LPAREN expression RPAREN

    Terminals, with rules where they appear

    TIMES                : 3
    error                : 
    MINUS                : 2
    RPAREN               : 6
    LPAREN               : 6
    DIVIDE               : 4
    PLUS                 : 1
    NUMBER               : 5

    Nonterminals, with rules where they appear

    expression           : 1 1 2 2 3 3 4 4 6 0


    Parsing method: LALR


    state 0

        S' -> . expression
        expression -> . expression PLUS expression
        expression -> . expression MINUS expression
        expression -> . expression TIMES expression
        expression -> . expression DIVIDE expression
        expression -> . NUMBER
        expression -> . LPAREN expression RPAREN

        NUMBER          shift and go to state 3
        LPAREN          shift and go to state 2


    state 1

        S' -> expression .
        expression -> expression . PLUS expression
        expression -> expression . MINUS expression
        expression -> expression . TIMES expression
        expression -> expression . DIVIDE expression

        PLUS            shift and go to state 6
        MINUS           shift and go to state 5
        TIMES           shift and go to state 4
        DIVIDE          shift and go to state 7


    state 2

        expression -> LPAREN . expression RPAREN
        expression -> . expression PLUS expression
        expression -> . expression MINUS expression
        expression -> . expression TIMES expression
        expression -> . expression DIVIDE expression
        expression -> . NUMBER
        expression -> . LPAREN expression RPAREN

        NUMBER          shift and go to state 3
        LPAREN          shift and go to state 2


    state 3

        expression -> NUMBER .

        $               reduce using rule 5
        PLUS            reduce using rule 5
        MINUS           reduce using rule 5
        TIMES           reduce using rule 5
        DIVIDE          reduce using rule 5
        RPAREN          reduce using rule 5


    state 4

        expression -> expression TIMES . expression
        expression -> . expression PLUS expression
        expression -> . expression MINUS expression
        expression -> . expression TIMES expression
        expression -> . expression DIVIDE expression
        expression -> . NUMBER
        expression -> . LPAREN expression RPAREN

        NUMBER          shift and go to state 3
        LPAREN          shift and go to state 2


    state 5

        expression -> expression MINUS . expression
        expression -> . expression PLUS expression
        expression -> . expression MINUS expression
        expression -> . expression TIMES expression
        expression -> . expression DIVIDE expression
        expression -> . NUMBER
        expression -> . LPAREN expression RPAREN

        NUMBER          shift and go to state 3
        LPAREN          shift and go to state 2


    state 6

        expression -> expression PLUS . expression
        expression -> . expression PLUS expression
        expression -> . expression MINUS expression
        expression -> . expression TIMES expression
        expression -> . expression DIVIDE expression
        expression -> . NUMBER
        expression -> . LPAREN expression RPAREN

        NUMBER          shift and go to state 3
        LPAREN          shift and go to state 2


    state 7

        expression -> expression DIVIDE . expression
        expression -> . expression PLUS expression
        expression -> . expression MINUS expression
        expression -> . expression TIMES expression
        expression -> . expression DIVIDE expression
        expression -> . NUMBER
        expression -> . LPAREN expression RPAREN

        NUMBER          shift and go to state 3
        LPAREN          shift and go to state 2


    state 8

        expression -> LPAREN expression . RPAREN
        expression -> expression . PLUS expression
        expression -> expression . MINUS expression
        expression -> expression . TIMES expression
        expression -> expression . DIVIDE expression

        RPAREN          shift and go to state 13
        PLUS            shift and go to state 6
        MINUS           shift and go to state 5
        TIMES           shift and go to state 4

```

--------------------------------

### Specify Starting Symbol for Grammar

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

The starting symbol of a grammar, normally the first rule defined, can be explicitly set using a 'start' specifier or as an argument to the yacc() function.

```python
start = 'foo'
```

```python
parser = yacc.yacc(start='foo')
```

--------------------------------

### Define Arithmetic Expression Grammar with PLY Yacc

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

This example demonstrates defining a grammar for simple arithmetic expressions using PLY's yacc module. It requires a lexer to provide tokens. The parser is built using yacc.yacc() and then used to parse input strings.

```python
import ply.yacc as yacc

# Get the token map from the lexer.  This is required.
from calclex import tokens

def p_expression_plus(p):
    'expression : expression PLUS term'
    p[0] = p[1] + p[3]

def p_expression_minus(p):
    'expression : expression MINUS term'
    p[0] = p[1] - p[3]

def p_expression_term(p):
    'expression : term'
    p[0] = p[1]

def p_term_times(p):
    'term : term TIMES factor'
    p[0] = p[1] * p[3]

def p_term_div(p):
    'term : term DIVIDE factor'
    p[0] = p[1] / p[3]

def p_term_factor(p):
    'term : factor'
    p[0] = p[1]

def p_factor_num(p):
    'factor : NUMBER'
    p[0] = p[1]

def p_factor_expr(p):
    'factor : LPAREN expression RPAREN'
    p[0] = p[2]

# Error rule for syntax errors
def p_error(p):
    print("Syntax error in input!")

# Build the parser
parser = yacc.yacc()

while True:
   try:
       s = input('calc > ')
   except EOFError:
       break
   if not s: continue
   result = parser.parse(s)
   print(result)
```

--------------------------------

### Get Line Span and Lexing Span of a Symbol

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

When tracking is enabled, `p.linespan(num)` returns a tuple of (startline, endline) and `p.lexspan(num)` returns a tuple of (start, end) positions for a symbol.

```python
def p_expression(p):
    'expression : expression PLUS expression'
    p.lineno(1)        # Line number of the left expression
    p.lineno(2)        # line number of the PLUS operator
    p.lineno(3)        # line number of the right expression
    ...
    start,end = p.linespan(3)    # Start,end lines of the right expression
    starti,endi = p.lexspan(3)   # Start,end positions of right expression
```

--------------------------------

### Error Recovery: Synchronizing on Special Characters

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Return a token to the parser to synchronize error recovery, useful for special characters. This example reads ahead for a terminating ';'.

```python
def p_error(p):
    # Read ahead looking for a terminating ";"
    while True:
        tok = parser.token()             # Get the next token
        if not tok or tok.type == 'SEMI': break
    parser.errok()

    # Return SEMI to the parser as the next lookahead token
    return tok  
```

--------------------------------

### Embedded Action Example in PLY

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Demonstrates how to use an empty rule for embedded actions to execute code during intermediate parsing stages. Access grammar symbols using p[-1] and assign values to p[0].

```python
def p_foo(p):
    """foo : A seen_A B C D"
    print("Parsed a foo", p[1],p[3],p[4],p[5])
    print("seen_A returned", p[2])

def p_seen_A(p):
    "seen_A :"
    print("Saw an A = ", p[-1])   # Access grammar symbol to left
    p[0] = some_value            # Assign value to seen_A
```

--------------------------------

### Report Error Using Token Line Number

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

For error reporting, you can directly use `p.lineno(token_index)` on a specific token within a grammar rule, such as the `LPAREN` token in this example.

```python
def p_bad_func(p):
    'funccall : fname LPAREN error RPAREN'
    # Line number reported from LPAREN token
    print("Bad function call at line", p.lineno(2))
```

--------------------------------

### Example of Reduce/Reduce Conflict

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

This grammar snippet illustrates a potential reduce/reduce conflict between an assignment rule and an expression rule when parsing a number. PLY will typically warn about such conflicts and resolve them by choosing the rule that appears first.

```python
assignment :  ID EQUALS NUMBER
           |  ID EQUALS expression

expression : expression PLUS expression
           | expression MINUS expression
           | expression TIMES expression
           | expression DIVIDE expression
           | LPAREN expression RPAREN
           | NUMBER
```

--------------------------------

### Return to Initial State

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Use `begin()` to switch back to the 'INITIAL' lexing state.

```python
def t_foo_end(t):
    r'end_foo'
    t.lexer.begin('INITIAL')        # Back to the initial state
```

--------------------------------

### Get Line Number and Lexing Position of a Token

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Use `p.lineno(num)` to get the line number and `p.lexpos(num)` to get the lexing position for a specific symbol in a grammar rule.

```python
def p_expression(p):
    'expression : expression PLUS expression'
    line   = p.lineno(2)        # line number of the PLUS token
    index  = p.lexpos(2)        # Position of the PLUS token
```

--------------------------------

### Switch to a Lexer State

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Use the `begin()` method on the lexer to switch to a different lexing state.

```python
def t_begin_foo(t):
    r'start_foo'
    t.lexer.begin('foo')             # Starts 'foo' state
```

--------------------------------

### Computing Token Column Position

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Calculate the column position of a token by finding the start of the current line. This is useful for error reporting and can be computed on demand.

```python
# Compute column.
# 	input is the input text string
# 	token is a token instance
def find_column(input, token):
    line_start = input.rfind('\n', 0, token.lexpos) + 1
    return (token.lexpos - line_start) + 1
```

--------------------------------

### AST Construction with PLY (Class-Based)

Source: https://context7.com/dabeaz/ply/llms.txt

Demonstrates building typed AST nodes using classes for grammar rule actions. Supports downstream analysis passes.

```python
import ply.lex as lex
import ply.yacc as yacc

# --- AST Node Classes ---
class Node:
    pass

class BinOp(Node):
    def __init__(self, op, left, right):
        self.op, self.left, self.right = op, left, right
    def __repr__(self):
        return f"BinOp({self.op!r}, {self.left!r}, {self.right!r})"

class Num(Node):
    def __init__(self, value):
        self.value = value
    def __repr__(self):
        return f"Num({self.value})"

class Assign(Node):
    def __init__(self, name, value):
        self.name, self.value = name, value
    def __repr__(self):
        return f"Assign({self.name!r}, {self.value!r})"

# --- Lexer ---
tokens = ('NUMBER', 'NAME', 'PLUS', 'TIMES', 'EQUALS', 'SEMI')
t_PLUS   = r'\'
'
t_TIMES  = r'\*'
t_EQUALS = r'='
t_SEMI   = r';'
t_NAME   = r'[a-zA-Z_][a-zA-Z0-9_]*'
t_ignore = ' 	\n'

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

def t_error(t): t.lexer.skip(1)
lexer = lex.lex()

# --- Parser ---
precedence = (
    ('left', 'PLUS'),
    ('left', 'TIMES'),
)

stmts = []

def p_program(p):
    '''program : program stmt
               | stmt'''

def p_stmt_assign(p):
    'stmt : NAME EQUALS expr SEMI'
    stmts.append(Assign(p[1], p[3]))

def p_stmt_expr(p):
    'stmt : expr SEMI'
    stmts.append(p[1])

def p_expr_binop(p):
    '''expr : expr PLUS  expr
            | expr TIMES expr'''
    p[0] = BinOp(p[2], p[1], p[3])

def p_expr_name(p):
    'expr : NAME'
    p[0] = p[1]

def p_expr_number(p):
    'expr : NUMBER'
    p[0] = Num(p[1])

def p_error(p):
    raise SyntaxError(f"Unexpected: {p}")

parser = yacc.yacc()
parser.parse("x = 1 + 2 * 3; x + 10;")
for node in stmts:
    print(node)
# Output:
# Assign('x', BinOp('+', Num(1), BinOp('*', Num(2), Num(3))))
# BinOp('+', 'x', Num(10))
```

--------------------------------

### Defining Literal Characters

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Specify literal characters that should be returned as is by defining a `literals` variable. These are checked after regular expression rules and take precedence if they start a rule.

```python
literals = [ '+','-','*','/' ]
```

```python
literals = "+-*/"
```

--------------------------------

### Build and Use a PLY Parser

Source: https://github.com/dabeaz/ply/blob/master/README.md

Build a parser using the yacc() function and then use it to parse an input string. The result is an Abstract Syntax Tree (AST) representing the parsed expression.

```python
# Build the parser
parser = yacc()

# Parse an expression
ast = parser.parse('2 * 3 + 4 * (5 - x)')
print(ast)
```

--------------------------------

### Build Lexer from Environment

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Builds the lexer from the current environment. Ensure only a single lexer is defined per module to avoid validation errors.

```python
# Build the lexer from my environment and return it
        return lex.lex()
```

--------------------------------

### g.build_lritems

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Calculates all of the LR items for all productions in the grammar. This step is required before using the grammar for any kind of table generation.

```APIDOC
## g.build_lritems()

### Description
Calculates all of the LR items for all productions in the grammar. This step is required before using the grammar for any kind of table generation.
```

--------------------------------

### Enable Lexer Debugging

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Run lex.lex() with debug=True to generate detailed debugging information, including rules, master regular expressions, and token generation.

```python
lexer = lex.lex(debug=True)
```

--------------------------------

### Lexer Start Conditions (States) in PLY

Source: https://context7.com/dabeaz/ply/llms.txt

Declare multiple lexer states to activate different token rules based on context. The 'comment' state is exclusive, meaning only its rules apply while active.

```python
import ply.lex as lex

tokens = ('TEXT', 'COMMENT')

# Declare states: 'comment' is exclusive (only its rules apply while active)
states = (
    ('comment', 'exclusive'),
)

# INITIAL state: match the start of a block comment
def t_begin_comment(t):
    r'/'
    t.lexer.comment_start = t.lexer.lexpos
    t.lexer.begin('comment')          # switch to 'comment' state

# 'comment' state: accumulate until closing */
def t_comment_end(t):
    r'*/'
    t.type  = 'COMMENT'
    t.value = t.lexer.lexdata[t.lexer.comment_start - 2 : t.lexer.lexpos]
    t.lexer.begin('INITIAL')          # return to normal state
    return t

def t_comment_body(t):
    r'(.|\n)+'                         # consume everything else
    pass

# Normal text token
def t_TEXT(t):
    r'[^/]+'
    return t

t_comment_ignore = ''
t_ignore = ''

def t_error(t):      t.lexer.skip(1)
def t_comment_error(t): t.lexer.skip(1)

lexer = lex.lex()
lexer.input("hello /* this is a comment */ world")
for tok in lexer:
    print(tok)
```

--------------------------------

### Push a Lexer State onto the Stack

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Use `push_state()` to enter a new lexing state, saving the current state on a stack.

```python
def t_begin_foo(t):
    r'start_foo'
    t.lexer.push_state('foo')             # Starts 'foo' state
```

--------------------------------

### Build LR Items

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Calculates all LR items for grammar productions. This step is required before table generation.

```python
g.build_lritems()
```

--------------------------------

### Grammar Class Initialization

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Initializes a Grammar object with a list of terminal symbols.

```APIDOC
## Grammar(terminals)

### Description
Creates a new grammar object.

### Parameters
* **terminals** (list of strings) - A list of strings specifying the terminals for the grammar.
```

--------------------------------

### AST Construction using Tuples/Lists

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Construct AST nodes by returning tuples or lists from grammar rule functions. This is a minimal approach suitable for simpler trees.

```python
def p_expression_binop(p):
    '''expression : expression PLUS expression
                  | expression MINUS expression
                  | expression TIMES expression
                  | expression DIVIDE expression'''

    p[0] = ('binary-expression',p[2],p[1],p[3])

def p_expression_group(p):
    'expression : LPAREN expression RPAREN'
    p[0] = ('group-expression',p[2])

def p_expression_number(p):
    'expression : NUMBER'
    p[0] = ('number-expression',p[1])
```

--------------------------------

### Compute Follow Sets

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Computes all follow sets for non-terminals. Returns a dictionary mapping non-terminal names to possible following symbols.

```python
g.compute_follow()
```

--------------------------------

### Feed and Consume Tokens with lexer.input() / lexer.token()

Source: https://context7.com/dabeaz/ply/llms.txt

Feed an input string to a lexer and retrieve tokens one by one. The `token()` method returns `None` at the end of the input. Useful for iterating through tokens.

```python
import ply.lex as lex

tokens = ('NUMBER', 'PLUS', 'MINUS')
t_PLUS  = r'\+'
t_MINUS = r'-'
t_ignore = ' \t'

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

def t_error(t):
    t.lexer.skip(1)

lexer = lex.lex()

data = "10 + 20 - 5"
lexer.input(data)

while True:
    tok = lexer.token()
    if tok is None:
        break
    # Each LexToken has: .type (str), .value (any), .lineno (int), .lexpos (int)
    print(f"type={tok.type!r:10s} value={tok.value!r:5} line={tok.lineno} pos={tok.lexpos}")
```

--------------------------------

### Access Production and its LR Items

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

After building LR items, you can access a specific production by its index and then view its associated LR items.

```python
>>> p = g[1]
>>> p
Production(statement -> ID = expr)
```

```python
>>> p.lr_items
[LRItem(statement -> . ID = expr), 
 LRItem(statement -> ID . = expr), 
 LRItem(statement -> ID = . expr), 
 LRItem(statement -> ID = expr .)]
```

--------------------------------

### Build a PLY Lexer

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Initialize a PLY lexer instance using lex.lex(). This function automatically discovers token rules from the calling context.

```python
lexer = lex.lex()
```

--------------------------------

### Arithmetic Expression Grammar with Actions

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

This specification extends the basic arithmetic grammar with semantic actions for a simple calculator. Actions define how to compute values for grammar rules.

```bnf
Grammar                             Action
    --------------------------------    -------------------------------------------- 
    expression0 : expression1 + term    expression0.val = expression1.val + term.val
                | expression1 - term    expression0.val = expression1.val - term.val
                | term                  expression0.val = term.val

    term0       : term1 * factor        term0.val = term1.val * factor.val
                | term1 / factor        term0.val = term1.val / factor.val
                | factor                term0.val = factor.val

    factor      : NUMBER                factor.val = int(NUMBER.lexval)
                | ( expression )        factor.val = expression.val
```

--------------------------------

### Managing State with a Lexer Class

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Define a lexer as a class to encapsulate state. Instance attributes are used to store and manage state, and the `build` method is used to initialize the lexer.

```python
class MyLexer:
    ...
    def t_NUMBER(self,t):
        r'\d+'
        self.num_count += 1
        t.value = int(t.value)
        return t

    def build(self, **kwargs):
        self.lexer = lex.lex(object=self,**kwargs)

    def __init__(self):
        self.num_count = 0
```

--------------------------------

### Compute First Sets

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Computes all first sets for symbols in the grammar. Returns a dictionary mapping symbol names to their first symbols.

```python
g.compute_first()
```

--------------------------------

### yacc.yacc()

Source: https://context7.com/dabeaz/ply/llms.txt

Builds a parser by scanning the calling module for functions prefixed with `p_` that define grammar rules. It constructs LALR(1) parsing tables and returns a `Parser` object.

```APIDOC
## `yacc.yacc()` — Build a Parser

Scans the calling module for `p_`-prefixed functions whose docstrings define BNF grammar rules, constructs LALR(1) parsing tables, and returns a `Parser` object.

```python
import ply.lex as lex
import ply.yacc as yacc

# --- Lexer ---
tokens = ('NUMBER', 'PLUS', 'MINUS', 'TIMES', 'DIVIDE', 'LPAREN', 'RPAREN')

t_PLUS   = r'\+'
t_MINUS  = r'-'
t_TIMES  = r'\*'
t_DIVIDE = r'/'
t_LPAREN = r'\('
t_RPAREN = r'\)'
t_ignore = ' \t'

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

def t_error(t):
    print(f"Illegal character: {t.value[0]!r}")
    t.lexer.skip(1)

lexer = lex.lex()

# --- Parser ---
# precedence: listed low-to-high; associativity and level resolve shift/reduce conflicts
precedence = (
    ('left',  'PLUS', 'MINUS'),
    ('left',  'TIMES', 'DIVIDE'),
    ('right', 'UMINUS'),           # fictitious token for unary minus
)

def p_expression_binop(p):
    '''expression : expression PLUS   expression
                  | expression MINUS  expression
                  | expression TIMES  expression
                  | expression DIVIDE expression'''
    ops = {'+': lambda a, b: a + b, '-': lambda a, b: a - b,
           '*': lambda a, b: a * b, '/': lambda a, b: a / b}
    p[0] = ops[p[2]](p[1], p[3])

def p_expression_uminus(p):
    'expression : MINUS expression %prec UMINUS'
    p[0] = -p[2]

def p_expression_group(p):
    'expression : LPAREN expression RPAREN'
    p[0] = p[2]

def p_expression_number(p):
    'expression : NUMBER'
    p[0] = p[1]

def p_error(p):
    if p:
        print(f"Syntax error at {p.value!r} (line {p.lineno})")
    else:
        print("Syntax error at EOF")

parser = yacc.yacc()

# Parse expressions
print(parser.parse("3 + 4 * 2"))          # 11
print(parser.parse("(3 + 4) * 2"))        # 14
print(parser.parse("-5 * (2 + 3)"))       # -25
print(parser.parse("10 / 2 - 1"))         # 4.0
```
```

--------------------------------

### Define and Use Empty Productions

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Empty productions can be handled by defining a rule with an empty right-hand side. Using a dedicated 'empty' rule symbol improves readability.

```python
def p_empty(p):
    'empty :'
    pass
```

```python
def p_optitem(p):
    'optitem : item'
    '        | empty'
```

--------------------------------

### Enable Debugging for lex() and yacc()

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Enable debugging for lexer and parser creation by setting the 'debug' flag to True. This is useful for diagnosing issues during parser and lexer generation.

```python
lex.lex(debug=True)
yacc.yacc(debug=True)
```

--------------------------------

### lexer.input() / lexer.token() — Feed and Consume Tokens

Source: https://context7.com/dabeaz/ply/llms.txt

Feeds an input string into a built lexer and retrieves tokens one at a time. `token()` returns `None` at end-of-input. Each `LexToken` has `.type`, `.value`, `.lineno`, and `.lexpos` attributes.

```APIDOC
## lexer.input() / lexer.token() — Feed and Consume Tokens

Feeds an input string into a built lexer and retrieves tokens one at a time. `token()` returns `None` at end-of-input.

```python
import ply.lex as lex

tokens = ('NUMBER', 'PLUS', 'MINUS')
t_PLUS  = r'\+'
t_MINUS = r'-'
t_ignore = ' \t'

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

def t_error(t):
    t.lexer.skip(1)

lexer = lex.lex()

data = "10 + 20 - 5"
lexer.input(data)

while True:
    tok = lexer.token()
    if tok is None:
        break
    # Each LexToken has: .type (str), .value (any), .lineno (int), .lexpos (int)
    print(f"type={tok.type!r:10s} value={tok.value!r:5} line={tok.lineno} pos={tok.lexpos}")

# Output:
# type='NUMBER'    value=10    line=1 pos=0
# type='PLUS'      value='+'   line=1 pos=3
# type='NUMBER'    value=20    line=1 pos=5
# type='MINUS'     value='-'   line=1 pos=8
# type='NUMBER'    value=5     line=1 pos=10
```
```

--------------------------------

### lex.lex() — Build a Lexer

Source: https://context7.com/dabeaz/ply/llms.txt

This function scans the calling module for `t_`-prefixed names and functions, compiles them into a master regular expression, and returns a `Lexer` object ready to tokenize input. It requires a `tokens` tuple and defines rules for tokenization.

```APIDOC
## lex.lex() — Build a Lexer

Scans the calling module for `t_`-prefixed names and functions, compiles them into a master regular expression, and returns a `Lexer` object ready to tokenize input.

```python
import ply.lex as lex

# Required: declare every token name used by the lexer (and later by yacc)
tokens = (
    'NUMBER', 'PLUS', 'MINUS', 'TIMES', 'DIVIDE', 'LPAREN', 'RPAREN', 'ID',
    'IF', 'ELSE', 'WHILE',
)

# Reserved words: handled inside a single ID rule to avoid prefix-matching bugs
reserved = {'if': 'IF', 'else': 'ELSE', 'while': 'WHILE'}

# Simple string rules (sorted longest-first automatically)
t_PLUS   = r'\+'
t_MINUS  = r'-'
t_TIMES  = r'\*'
t_DIVIDE = r'/'
t_LPAREN = r'\('
t_RPAREN = r'\)'

# Characters to skip entirely (very fast, handled before regex matching)
t_ignore = ' \t'

# Function rule: docstring IS the regex; action code transforms the token value
def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)   # convert matched string to int
    return t

# ID rule that also catches reserved words
def t_ID(t):
    r'[a-zA-Z_][a-zA-Z0-9_]*'
    t.type = reserved.get(t.value, 'ID')   # remap to reserved word type if needed
    return t

# Track line numbers (token is discarded — no return)
def t_newline(t):
    r'\n+'
    t.lexer.lineno += len(t.value)

# Required error handler: skip the bad character and keep going
def t_error(t):
    print(f"Illegal character {t.value[0]!r} at line {t.lexer.lineno}")
    t.lexer.skip(1)

# Build and use the lexer
lexer = lex.lex()
lexer.input("if x + 42 * (y - 3)")

for tok in lexer:
    print(tok)
# Output:
# LexToken(IF,'if',1,0)
# LexToken(ID,'x',1,3)
# LexToken(PLUS,'+',1,5)
# LexToken(NUMBER,42,1,7)
# LexToken(TIMES,'*',1,10)
# LexToken(LPAREN,'(',1,12)
# LexToken(ID,'y',1,13)
# LexToken(MINUS,'-',1,14)
# LexToken(NUMBER,3,1,16)
# LexToken(RPAREN,')',1,17)
```
```

--------------------------------

### Supplying an Alternative Tokenizer in PLY

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Demonstrates how to provide a custom tokenizer to `yacc.parse()` by passing a Lexer object with a `token()` method. The lexer must also have an `input()` method if `yacc.parse()` is called with an input string.

```python
parser = yacc.parse(lexer=x)
```

--------------------------------

### Use PLY Lexer's Main Function for Input

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Include the if __name__ == '__main__': block with lex.runmain() to enable tokenizing input from standard input or a specified file.

```python
if __name__ == '__main__':
     lex.runmain()
```

--------------------------------

### AST Construction using Custom Classes

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Build AST nodes using custom classes for better organization and extensibility. This approach is beneficial for adding semantics, type checking, or code generation.

```python
class Expr: pass

class BinOp(Expr):
    def __init__(self,left,op,right):
        self.left = left
        self.right = right
        self.op = op

class Number(Expr):
    def __init__(self,value):
        self.value = value

def p_expression_binop(p):
    '''expression : expression PLUS expression
                  | expression MINUS expression
                  | expression TIMES expression
                  | expression DIVIDE expression'''

    p[0] = BinOp(p[1],p[2],p[3])

def p_expression_group(p):
    'expression : LPAREN expression RPAREN'
    p[0] = p[2]

def p_expression_number(p):
    'expression : NUMBER'
    p[0] = Number(p[1])
```

--------------------------------

### Define Tokens in All States

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Define a token to be active in all lexer states using the 'ANY' keyword.

```python
t_ANY_NUMBER = r'\d+'         # Defines a token 'NUMBER' in all states
```

--------------------------------

### Build LR Items for a Grammar

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Call `Grammar.build_lritems()` to attach LR items to all productions in the grammar. This is a prerequisite for inspecting LR item details.

```python
>>> g.build_lritems()
```

--------------------------------

### g.compute_first

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

Computes all of the first sets for all symbols in the grammar.

```APIDOC
## g.compute_first()

### Description
Compute all of the first sets for all symbols in the grammar.

### Returns
* dict - A dictionary mapping symbol names to a list of all first symbols.
```

--------------------------------

### Inspect LRItem Attributes

Source: https://github.com/dabeaz/ply/blob/master/doc/internals.md

An LRItem instance provides details about a specific parsing stage, including the rule name, production symbols, dot position, and possible next symbols.

```python
>>> lr
LRItem(statement -> ID = . expr)
```

```python
>>> lr.lr_after
[Production(expr -> expr PLUS expr), 
 Production(expr -> expr MINUS expr), 
 Production(expr -> expr TIMES expr), 
 Production(expr -> expr DIVIDE expr), 
 Production(expr -> MINUS expr), 
 Production(expr -> LPAREN expr RPAREN), 
 Production(expr -> NUMBER), 
 Production(expr -> ID)]
```

--------------------------------

### Import PLY Lex and Yacc Modules

Source: https://github.com/dabeaz/ply/blob/master/README.md

Import the lex and yacc modules from the PLY package. These files can be copied directly into your project.

```python
from .ply import lex
from .ply import yacc
```

--------------------------------

### Tokenize Input Data with PLY Lexer

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

This code demonstrates how to feed input data to a PLY lexer and process the generated tokens. It uses a while loop to repeatedly call the token() method until no more tokens are available.

```python
# Test it out
data = '''
3 + 4 * 10
  + -20 *2
'''

# Give the lexer some input
lexer.input(data)

# Tokenize
while True:
    tok = lexer.token()
    if not tok: 
        break      # No more input
    print(tok)
```

--------------------------------

### Enable Automatic Line Number and Position Tracking

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Enable automatic tracking for all grammar symbols by passing `tracking=True` to `yacc.parse()`. This allows access to `lineno()`, `lexpos()`, `linespan()`, and `lexspan()` for all symbols.

```python
yacc.parse(data,tracking=True)
```

--------------------------------

### Build AST as Nested Tuples

Source: https://context7.com/dabeaz/ply/llms.txt

Defines grammar rules to construct an Abstract Syntax Tree (AST) represented as nested tuples. This approach is useful for simple expression parsing where a direct AST representation is desired.

```python
def p_expr_binop(p):
    '''expr : expr PLUS  expr
            | expr TIMES expr'''
    p[0] = (p[2], p[1], p[3])

def p_expr_group(p):
    'expr : LPAREN expr RPAREN'
    p[0] = p[2]

def p_expr_number(p):
    'expr : NUMBER'
    p[0] = ('num', p[1])

def p_error(p):
    raise SyntaxError(f"Unexpected token: {p}")

parser = yacc.yacc()

# Basic parse — returns the top-level p[0] value
ast = parser.parse("2 + 3 * 4")
print(ast)
# ('+', ('num', 2), ('*', ('num', 3), ('num', 4)))

# Parse with position tracking enabled
ast2 = parser.parse("(1 + 2) * 3", tracking=True)
print(ast2)
# ('*', ('+', ('num', 1), ('num', 2)), ('num', 3))

# Reuse a specific lexer instance (important with multiple lexers)
my_lexer = lex.lex()
result = parser.parse("10 + 5", lexer=my_lexer)
print(result)
# ('+', ('num', 10), ('num', 5))
```

--------------------------------

### Tracking Source Location Data with PLY

Source: https://context7.com/dabeaz/ply/llms.txt

Enables precise error messages by recording source location data for grammar symbols. Use `tracking=True` to enable linespan/lexspan for non-terminal symbols.

```python
import ply.lex as lex
import ply.yacc as yacc

tokens = ('NUMBER', 'PLUS', 'MINUS', 'TIMES', 'NEWLINE')
t_PLUS   = r'\'
'
t_MINUS  = r'-'
t_TIMES  = r'\*'
t_ignore = ' 	'

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

def t_NEWLINE(t):
    r'\n'
    t.lexer.lineno += 1
    return t

def t_error(t): t.lexer.skip(1)
lexer = lex.lex()

errors = []

def p_program(p):
    '''program : program stmt NEWLINE
               | stmt NEWLINE'''

def p_stmt(p):
    '''stmt : expr PLUS  expr
            | expr MINUS expr
            | expr TIMES expr
            | expr'''
    if len(p) == 4:
        line_of_op = p.lineno(2)     # line number of the operator token
        pos_of_op  = p.lexpos(2)     # character offset of the operator
        p[0] = (p[2], p[1], p[3], line_of_op, pos_of_op)
    else:
        p[0] = p[1]

def p_expr(p):
    'expr : NUMBER'
    p[0] = p[1]

def p_error(p):
    if p:
        errors.append(f"Line {p.lineno}, pos {p.lexpos}: unexpected {p.value!r}")

parser = yacc.yacc()

# tracking=True enables linespan/lexspan for non-terminal symbols too
src = "1 + 2\n3 * 4\n"
parser.parse(src, lexer=lexer, tracking=True)

# Manually check position for a token
lexer.input("hello + world")
tok = lexer.token()
source = "hello + world"
line_start = source.rfind('\n', 0, tok.lexpos) + 1
col = (tok.lexpos - line_start) + 1
print(f"Token {tok.value!r} at column {col}")  # Token 'hello' at column 1
```

--------------------------------

### Pop a Lexer State from the Stack

Source: https://github.com/dabeaz/ply/blob/master/doc/ply.md

Use `pop_state()` to return to the previous lexing state from the stack.

```python
def t_foo_end(t):
    r'end_foo'
    t.lexer.pop_state()                   # Back to the previous state
```