### Lexer Lookup Examples

Source: https://pygments.org/docs/quickstart

Shows how to get lexers by name, filename, or MIME type.

```python
>>> from pygments.lexers import (
...     get_lexer_by_name,
...     get_lexer_for_filename,
...     get_lexer_for_mimetype)

>>> get_lexer_by_name('python')
<pygments.lexers.PythonLexer>

>>> get_lexer_for_filename('spam.rb')
<pygments.lexers.RubyLexer>

>>> get_lexer_for_mimetype('text/x-perl')
<pygments.lexers.PerlLexer>

```

--------------------------------

### Lexer Constructor Example

Source: https://pygments.org/docs/api

Example of how to implement a custom lexer's constructor, processing its own options and then calling the base Lexer constructor.

```python
def __init__(self, **options):
    self.compress = options.get('compress', '')
    Lexer.__init__(self, **options)
```

--------------------------------

### Output Lexer Example

Source: https://pygments.org/docs/lexers

Simple lexer that highlights everything as Token.Generic.Output. Added in version 2.10.

```text
,                                
        \`-._           __                
        \\  `-..____,.'  `.              
        :`.         /    \`.            
        :  )       :      : \           
        ;'        '   ;  |  :          
        )..      .. .:.`.;  :          
        /::...  .:::...   ` ;          
        ; _ '    __        /:\         
        `:o>   /\o_>      ;:. `.        
        `-`.__ ;   __..--- /:.   \       
        === \_/   ;=====_.':.     ;      
        ,/'`--'...`--....        ;      
            ;                    ;      
            .'                      ;      
        .'                        ;      
        .'     ..     ,      .       ;      
        :       ::..  /      ;::.     |      
    /      `.;::.  |       ;:..    ;      
    :         |:.   :       ;:.    ;      
    :         ::     ;:..   |.    ;      
    :       :;      :::....|     |      
    /\     ,/ \      ;:::::;     ;      
    .:. \:..|    :     ; '.--|     ;      
    ::.  :''  `-.,,;     ;'   ;     ;      
.-'. _.'\      / `;      \,__:      \     
`---'    `----'   ;      /    \,.,,,/
                    `----`              fsc

```

--------------------------------

### PostgreSQL EXPLAIN Output Lexer Example

Source: https://pygments.org/docs/lexers

Handles syntax highlighting for PostgreSQL EXPLAIN command output.

```postgresql-explain
=> EXPLAIN SELECT * FROM tenk1
-> WHERE unique1 < 100;   -- Don't take -> in the plan as a prompt

                                  QUERY PLAN
------------------------------------------------------------------------------
  Bitmap Heap Scan on tenk1  (cost=2.37..232.35 rows=106 width=244)
    Recheck Cond: (unique1 < 100)
    ->  Bitmap Index Scan on tenk1_unique1  (cost=0.00..2.37 rows=106 width=0)
          Index Cond: (unique1 < 100)

```

--------------------------------

### Highlight Python Code

Source: https://pygments.org/docs/quickstart

Basic example for highlighting Python code and printing the HTML output.

```python
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

code = 'print "Hello World"'
print(highlight(code, PythonLexer(), HtmlFormatter()))

```

--------------------------------

### Install Pygments with Plugin Support

Source: https://pygments.org/docs/changelog

Install Pygments with the 'plugins' extra to ensure optimal plugin support, especially for Python versions older than 3.8, by including the `importlib_metadata` backport.

```bash
pip install pygments[plugins]

```

--------------------------------

### Elixir Console Lexer Example

Source: https://pygments.org/docs/lexers

Demonstrates the output format for the Elixir interactive console (iex).

```elixir
iex> [head | tail] = [1,2,3]
[1,2,3]
iex> head
1
iex> tail
[2,3]
iex> [head | tail]
[1,2,3]
iex> length [head | tail]
3
```

--------------------------------

### Example Pygments HTML Output

Source: https://pygments.org/docs/java

This is an example of the HTML output generated by Pygments for syntax highlighting.

```html
<div class="highlight">
<pre><span class="k">print</span> <span class="s">&quot;Hello World&quot;</span></pre>
</div>
```

--------------------------------

### PostgreSQL Console Lexer Example

Source: https://pygments.org/docs/lexers

Demonstrates syntax highlighting for psql sessions, including commands, errors, and output formatting.

```postgresql-console
regression=# select foo;
ERROR:  column "foo" does not exist
CONTEXT:  PL/pgSQL function "test1" while casting return value to function's return type
LINE 1: select foo;
               ^
regression=# \q

peter@localhost testdb=> \a \t \x
Output format is aligned.
Tuples only is off.
Expanded display is on.

regression=# select '\x';
WARNING:  nonstandard use of escape in a string literal
LINE 1: select '\x';
               ^
HINT:  Use the escape string syntax for escapes, e.g., E'\r\n'.
 ?column? 
----------
 x
(1 row)

regression=# select E'\x';

piro=> \set foo 30;
piro=> select * from test where foo <= :foo;
 foo | bar 
-----+-----
  10 | 
  20 | 
(2 rows)

testdb=> \set foo 'my_table'
testdb=> SELECT * FROM :"foo";

testdb=> \set content `cat my_file.txt`
testdb=> INSERT INTO my_table VALUES (:'content');

regression=# select (
regression(# 1);
 ?column? 
----------
        1
(1 row)

piro=> select (
piro(> '' || $$
piro$> $$
piro-> from "
piro"> foo";
ERROR:  relation "
foo" does not exist
LINE 5: from "
             ^

testdb=> CREATE TABLE my_table (
first integer not null default 0,
second text) ; -- end of command
CREATE TABLE

-- Table output
=# SELECT '0x10'::mpz AS "hex", '10'::mpz AS "dec",
-#        '010'::mpz AS oct, '0b10'::mpz AS bin;
 hex | dec | oct | bin
-----+-----+-----+
 16  | 10  | 8   | 2
(1 row)

-- One field output
regression=# select schemaname from  pg_tables limit 3;
 schemaname 
------------
 pg_catalog
 pg_catalog
 pg_catalog
(3 rows)

-- TODO: prompt in multiline comments still not handled correctly
test=> select 1 /* multiline
test*> and 2 /* and 3 */
test*> end comment */, 2;
 ?column? | ?column? 
----------+----------
        1 |        2

=# select 10.0, 1e-6, 1E+6;
 ?column? | ?column? | ?column? 
----------+----------+----------
     10.0 | 0.000001 |  1000000
(1 row)

regression=# begin;
BEGIN
regression=# create table asdf (foo serial primary key);
NOTICE:  CREATE TABLE will create implicit sequence "asdf_foo_seq" for serial column "asdf.foo"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "asdf_pkey" for table "asdf"
CREATE TABLE
regression=# insert into asdf values (10) returning foo;
 foo 
-----
  10
(1 row)

INSERT 0 1
regression=# ROLLBACK ;
ROLLBACK

=> EXPLAIN SELECT * FROM tenk1
-> WHERE unique1 < 100;   -- Don't take -> in the plan as a prompt

                                  QUERY PLAN
------------------------------------------------------------------------------
  Bitmap Heap Scan on tenk1  (cost=2.37..232.35 rows=106 width=244)
    Recheck Cond: (unique1 < 100)
    ->  Bitmap Index Scan on tenk1_unique1  (cost=0.00..2.37 rows=106 width=0)
          Index Cond: (unique1 < 100)


-- don't swallow the end of a malformed line
test=> select 1,
'this line must be emitted'

```

--------------------------------

### List styles as JSON

Source: https://pygments.org/docs/cmdline

Combine the -L option with --json to output the list of installed styles and their descriptions in JSON format.

```bash
$ pygmentize -L styles --json

```

--------------------------------

### Get help for lexer, formatter, or filter

Source: https://pygments.org/docs/cmdline

Use the -H option to get detailed information about a specific lexer, formatter, or filter.

```bash
$ pygmentize -H formatter html

```

```bash
$ pygmentize -H lexer python

```

--------------------------------

### Get Lua Built-in Modules

Source: https://pygments.org/docs/lexers

Demonstrates how to retrieve a list of available built-in modules for Lua highlighting.

```python
>>> from pygments.lexers._lua_builtins import MODULES
>>> MODULES.keys()
['string', 'coroutine', 'modules', 'io', 'basic', ...]
```

--------------------------------

### GAP Console Lexer Example

Source: https://pygments.org/docs/lexers

Demonstrates the usage of the GAP Console Lexer with various mathematical operations and checks for different ring types.

```gap
#@local checkEuclideanRing
gap> START_TEST("euclidean.tst");

# test consistency of EuclideanDegree, EuclideanQuotient, EuclideanRemainder,
# and QuotientRemainder for some ring and elements of it
gap> checkEuclideanRing :=
> function(R, colls...)
>   local coll1, coll2, a, b, deg_b, deg_r, q, r, qr;
>   if Length(colls) >= 1 then coll1:=colls[1];
>   elif Size(R) <= 100 then coll1 := R;
>   else coll1 := List([1..100],i->Random(R));
>   fi;
>   if Length(colls) >= 2 then coll2:=colls[2];
>   elif Size(R) <= 100 then coll2 := R;
>   else coll2 := List([1..100],i->Random(R));
>   fi;
>   for b in coll1 do
>     if IsZero(b) then continue; fi;
>     deg_b := EuclideanDegree(R, b);
>     for a in coll2 do
>       q := EuclideanQuotient(R, a, b); Assert(0, q in R);
>       r := EuclideanRemainder(R, a, b); Assert(0, r in R);
>       if a <> q*b + r then Error("a <> q*b + r for ", [R,a,b]); fi;
>       deg_r := EuclideanDegree(R, r);
>       if not IsZero(r) and deg_r >= deg_b then Error("Euclidean degree did not decrease for ",[R,a,b]); fi;
>       qr := QuotientRemainder(R, a, b);
>       if qr <> [q, r] then Error("QuotientRemainder inconsistent for ", [R,a,b]); fi;
>     od;
>   od;
>   return true;
> end;;

# rings in characteristic 0
gap> checkEuclideanRing(Integers,[-100..100],[-100..100]);
true
gap> checkEuclideanRing(Rationals);
true
gap> checkEuclideanRing(GaussianIntegers);
true
gap> checkEuclideanRing(GaussianRationals);
true

# finite fields
gap> ForAll(Filtered([2..50], IsPrimePowerInt), q->checkEuclideanRing(GF(q)));
true

# ZmodnZ
gap> ForAll([1..50], m -> checkEuclideanRing(Integers mod m));
true
gap> checkEuclideanRing(Integers mod ((2*3*5)^2));
true
gap> checkEuclideanRing(Integers mod ((2*3*5)^3));
true
gap> checkEuclideanRing(Integers mod ((2*3*5*7)^2));
true
gap> checkEuclideanRing(Integers mod ((2*3*5*7)^3));
true

```

--------------------------------

### Use pygmentize to format a file

Source: https://pygments.org/docs/changelog

Example of using the `pygmentize` command-line tool to format a Python file into a GIF image.

```bash
pygmentize -f gif -o foo.gif foo.py

```

--------------------------------

### Get All Available Styles

Source: https://pygments.org/docs/styles

Use `get_all_styles` to get an iterable of all registered styles, including those from plugins. Convert to a list to see all names.

```python
>>> from pygments.styles import get_all_styles
>>> styles = list(get_all_styles())
```

--------------------------------

### PsySH Console Lexer Example

Source: https://pygments.org/docs/lexers

This lexer is for highlighting PsySH console output, demonstrating interactive PHP sessions.

```php
>>> $greeting = function($name): string {
...     return "Hello, {$name}";
... };
=> Closure($name): string {#2371 …3}
>>> $greeting('World')
=> "Hello, World"

```

--------------------------------

### Iterate All Lexers

Source: https://pygments.org/docs/lexers

Demonstrates how to get an iterator for all available lexers (builtin and plugin) using `get_all_lexers()`.

```python
>>> from pygments.lexers import get_all_lexers
>>> i = get_all_lexers()
>>> i.next()
('Diff', ('diff',), ('*.diff', '*.patch'), ('text/x-diff', 'text/x-patch'))
>>> i.next()
('Delphi', ('delphi', 'objectpascal', 'pas', 'pascal'), ('*.pas',), ('text/x-pascal',))
>>> i.next()
('XML+Ruby', ('xml+ruby', 'xml+erb'), (), ())
```

--------------------------------

### SQLite Interactive Session Lexer

Source: https://pygments.org/docs/lexers

Lexer for example sessions using sqlite3. This highlights commands and output from an interactive SQLite console.

```sqlite3
SQLite version 3.4.2
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE paste (paste_id integer, code text, parsed_code text, pub_date
varchar(24), language varchar(64), parent_id integer, url varchar(128));
CREATE TABLE vars (key varchar(24), value varchar(128));
sqlite> a '
   ...>
   ...>';
SQL error: near "a": syntax error
sqlite> %;
SQL error: near "%”: syntax error
sqlite> select count(language), language from paste group by language order
   ...> by count(language) desc;
144|python
76|text
22|pycon
9|ruby
7|c
7|js
6|html+django
4|html
4|tex
2|html+php
1|cpp
1|scheme
sqlite> 

```

--------------------------------

### List Available Styles

Source: https://pygments.org/docs/styles

Use `STYLE_MAP` to get a dictionary of available styles by name. This is useful for discovering styles.

```python
>>> from pygments.styles import STYLE_MAP
>>> STYLE_MAP.keys()
['default', 'emacs', 'friendly', 'colorful']
```

--------------------------------

### Julia Console Session Example

Source: https://pygments.org/docs/lexers

Demonstrates a typical interactive session in the Julia REPL, including variable assignment, multi-line string definition with embedded code, and function calls. Use this lexer for Julia console output.

```julia-repl
julia> x = 2
2

julia> """
        The factorial function.

        ```julia
        @assert fac(3) == 1 * 2 * 3
        ```
        """
        function fac(n)
            if n < 2
                return 1
            else
                return n * fac(n - 1) # <-- recursive call
            end
        end


        # Lets try the function out...
        f(x + 1)
6

```

--------------------------------

### Matlab Session Lexer Example

Source: https://pygments.org/docs/lexers

Demonstrates a typical interactive session in Matlab, including variable assignments, function calls, and error messages. This lexer is modeled after PythonConsoleLexer.

```matlab
>>
>>
>> a = 'okay'

a =

okay

>> x = rand(3)    % a matrix

x =

    0.8147    0.9134    0.2785
    0.9058    0.6324    0.5469
    0.1270    0.0975    0.9575

>> 1/0

ans =

   Inf

>> foo
??? Undefined function or variable 'foo'.

>>
>>
>> {cos(2*pi), 'testing'}

ans =

    [1]    'testing'

>>
>>
>>

```

--------------------------------

### Create POSIX Alias for Pygmentize

Source: https://pygments.org/docs/terminaltips

Define a shorthand command for `pygmentize`. This example creates an alias `cath` for basic highlighting. Add flags like `style=monokai` for specific themes.

```bash
$ alias cath="pygmentize"
```

```bash
$ alias cath="pygmentize -O style=monokai"
```

--------------------------------

### Configure Lexer and Formatter Options

Source: https://pygments.org/docs/quickstart

Demonstrates using options like stripall for lexers and linenos, cssclass for formatters.

```python
from pygments import highlight
from pygments.lexers import get_lexer_by_name
from pygments.formatters import HtmlFormatter

code = 'print "Hello World"'
lexer = get_lexer_by_name("python", stripall=True)
formatter = HtmlFormatter(linenos=True, cssclass="source")
result = highlight(code, lexer, formatter)

```

--------------------------------

### Dylan Interactive Console Lexer Example

Source: https://pygments.org/docs/lexers

Example usage of the Dylan interactive console lexer. This lexer is based on the RubyConsoleLexer.

```text
? let a = 1;
=> 1
? a
=> 1
```

--------------------------------

### Full HTML Document with Options

Source: https://pygments.org/docs/cmdline

Creates a full HTML document with line numbers and the 'emacs' style, highlighting test.py to test.html.

```bash
$ pygmentize -O full,style=emacs,linenos=1 -o test.html test.py

```

--------------------------------

### Loading Lexer from File (Command Line)

Source: https://pygments.org/docs/lexerdevelopment

Demonstrates how to load and use a custom lexer defined in a Python file from the Pygments command-line interface using the `-x` and `-l` flags.

```bash
$ python -m pygments -x -l your_lexer_file.py <inputfile>

```

--------------------------------

### Get All Styles

Source: https://pygments.org/docs/api

Returns a generator that yields the names of all available styles, including both built-in and plugin styles.

```APIDOC
## pygments.styles.get_all_styles()

### Description
Return a generator for all styles by name, both builtin and plugin.

### Returns
- generator - A generator yielding style names.
```

--------------------------------

### Generate Full HTML Document

Source: https://pygments.org/docs/quickstart

Use `pygmentize` with `-f html -O full` to generate a complete HTML document, including the stylesheet, for the highlighted code.

```bash
$ pygmentize -f html -O full -o test.html test.py
```

--------------------------------

### Applying Lexer and Formatter Options

Source: https://pygments.org/docs/cmdline

Applies 'colorful' style and line numbers to the HTML output of test.py using the Python lexer.

```bash
$ pygmentize -f html -O style=colorful,linenos=1 -l python test.py

```

--------------------------------

### Add Filter by String and Options

Source: https://pygments.org/docs/filters

Demonstrates adding a filter using its string name and passing options to its constructor.

```python
>>> from pygments.lexers import PythonLexer
>>> l = PythonLexer()
>>> # add a filter given by a string and options
>>> l.add_filter('codetagify', case='lower')
>>> l.filters
[<pygments.filters.CodeTagFilter object at 0xb785decc>]
```

--------------------------------

### Highlight File to HTML Output

Source: https://pygments.org/docs/quickstart

Use the `pygmentize` command with the `-f html` option to output highlighted code as HTML to a specified file.

```bash
$ pygmentize -f html -o test.html test.py
```

--------------------------------

### Get Style by Name

Source: https://pygments.org/docs/api

Retrieves a specific style class using its short name. Raises ClassNotFound if the style is not found.

```APIDOC
## pygments.styles.get_style_by_name(_name_)

### Description
Return a style class by its short name. The names of the builtin styles are listed in `pygments.styles.STYLE_MAP`.
Will raise `pygments.util.ClassNotFound` if no style of that name is found.

### Parameters
#### Path Parameters
- **_name_** (string) - Required - The short name of the style to retrieve.
```

--------------------------------

### Create and Access Token Types

Source: https://pygments.org/docs/tokens

Demonstrates how to create new token types by accessing attributes of `pygments.token.Token`. Tokens are singletons, allowing comparison with the `is` operator.

```python
>>> from pygments.token import Token
>>> Token.String
Token.String
>>> Token.String is Token.String
True
```

--------------------------------

### Generate Stylesheet

Source: https://pygments.org/docs/quickstart

Create a CSS stylesheet for Pygments by using the `pygmentize` command with `-S <style_name> -f html`.

```bash
$ pygmentize -S default -f html > style.css
```

--------------------------------

### Get Built-in Style by Name

Source: https://pygments.org/docs/styles

Use `get_style_by_name` to retrieve a specific built-in style class. This is useful for inspecting or directly using a style.

```python
>>> from pygments.styles import get_style_by_name
>>> get_style_by_name('colorful')
<class 'pygments.styles.colorful.ColorfulStyle'>
```

--------------------------------

### CustomLexer Boilerplate

Source: https://pygments.org/docs/lexerdevelopment

The minimal structure required for a custom lexer class inheriting from `RegexLexer`. This serves as a starting point for developing new lexers.

```python
from pygments.lexer import RegexLexer
from pygments.token import *

class CustomLexer(RegexLexer):
    """All your lexer code goes here!"""


```

--------------------------------

### Create Fish Shell Alias for Pygmentize

Source: https://pygments.org/docs/terminaltips

In fish shell, use functions to create permanent aliases. This example defines a `cath` function to alias `pygmentize`.

```fish
function cath
        # 'cath' alias will highlight source code as cat does.
        alias cath="pygmentize"
end
```

```fish
function cath
        alias cath="pygmentize -O style=monokai"
end
```

--------------------------------

### Use ISO dialect with p1 extensions via command line

Source: https://pygments.org/docs/lexers

Render Modula-2 source code using the ISO dialect with p1 extensions by specifying the dialect option.

```bash
$ pygmentize -O full,dialect=m2iso+p1 -f rtf -o /path/to/output /path/to/input
```

--------------------------------

### Convert Keyword Case

Source: https://pygments.org/docs/filters

The `keywordcase` filter converts keywords to lowercase, uppercase, or capitalizes them. This is useful for enforcing style guides, especially for languages like Pascal.

```python
from pygments.lexers import PythonLexer
from pygments.filters import KeywordCaseFilter

lexer = PythonLexer()
lexer.add_filter(KeywordCaseFilter(case='upper'))

code = "def my_function():\n    return True"

for token_type, token_value in lexer.get_tokens(code):
    print(f'{token_type}: {token_value}')
```

--------------------------------

### Disable rendering of standard library ADTs as builtins

Source: https://pygments.org/docs/lexers

Control the rendering of standard library first-class ADT identifiers in Modula-2 R10. This example disables rendering them as builtins.

```bash
$ pygmentize -O full,dialect=m2r10,treat_stdlib_adts_as_builtins=Off ...

```

--------------------------------

### Basic Usage: Highlight Python File

Source: https://pygments.org/docs/cmdline

Prints the file test.py to standard output, inferring the Python lexer and using the terminal formatter.

```bash
$ pygmentize test.py
print "Hello World"


```

--------------------------------

### Loading Lexer from File (Python API)

Source: https://pygments.org/docs/lexerdevelopment

Illustrates using the `load_lexer_from_file` function from the Pygments API to dynamically load custom lexers.

```python
# For a lexer named CustomLexer
your_lexer = load_lexer_from_file(filename, **options)

# For a lexer named MyNewLexer
your_named_lexer = load_lexer_from_file(filename, "MyNewLexer", **options)

```

--------------------------------

### Subclassing HtmlFormatter for Custom Wrapping

Source: https://pygments.org/docs/formatters

Demonstrates how to subclass HtmlFormatter to customize the wrapping of code blocks. This example wraps code in <code> tags and uses <br> for line breaks.

```python
class CodeHtmlFormatter(HtmlFormatter):

    def wrap(self, source, *, include_div):
        return self._wrap_code(source)

    def _wrap_code(self, source):
        yield 0, '<code>'
        for i, t in source:
            if i == 1:
                # it's a line of formatted code
                t += '<br>'
            yield i, t
        yield 0, '</code>'
```

--------------------------------

### Loading Specific Lexer Class from File (Command Line)

Source: https://pygments.org/docs/lexerdevelopment

Shows how to specify a lexer class name other than the default when loading a custom lexer from a file via the command line.

```bash
$ python -m pygments -x -l your_lexer.py:SomeLexer <inputfile>

```

--------------------------------

### Set Lexer Input Encoding with Chardet

Source: https://pygments.org/docs/unicode

Override the default input encoding for a lexer to automatically detect the correct encoding using the 'chardet' library. Ensure 'chardet' is installed.

```python
from pygments.lexers import PythonLexer
lexer = PythonLexer(encoding='chardet')
```

--------------------------------

### SourcePawn Lexer for SourcePawn

Source: https://pygments.org/docs/lexers

Use this lexer for SourcePawn source code, which includes preprocessor directives.

```python
from pygments.lexers import SourcePawnLexer

lexer = SourcePawnLexer()

```

--------------------------------

### HTML Formatter CSS Definitions (Single Prefix)

Source: https://pygments.org/docs/formatters

Shows how to get CSS rules for the HtmlFormatter's classes using a single prefix. This is useful for integrating Pygments styling into existing stylesheets.

```python
formatter.get_style_defs('td .code')
```

--------------------------------

### Hexdump Lexer Example

Source: https://pygments.org/docs/lexers

Highlights typical hex dump output formats from UNIX/Linux tools like hexdump, hd, hexcat, od, xxd, and the DOS DEBUG.exe tool. The canonical format (-C) is specifically supported.

```text
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  02 00 3e 00 01 00 00 00  c5 48 40 00 00 00 00 00  |..>......H@.....|
```

--------------------------------

### HTML Output with Inferred Formatter

Source: https://pygments.org/docs/cmdline

Generates HTML output for test.py, inferring the formatter from the output file extension and writing to test.html.

```bash
$ pygmentize -o test.html test.py

```

--------------------------------

### RegexLexer State Rule Example

Source: https://pygments.org/docs/lexerdevelopment

Illustrates a common pitfall in defining lexer states where the order of rules prevents later rules from ever being matched. This highlights the need for careful rule ordering or using helper functions like `bygroups`.

```python
'state': [
    (r'\w+', Name,),
    (r'\s+', Whitespace,),
    (r'\w+', Keyword,)
]
```