Doctrine Lexer

https://github.com/doctrine/lexer
Admin
Base library for a lexer that can be used in Top-Down, Recursive Descent Parsers, employed in...
Tokens:20,230
Snippets:137
Trust Score:9.7
License:MIT
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### peek() usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Example showing how to peek at the current and next tokens.

```php
$first = $lexer->peek();   // token at current position
$second = $lexer->peek();  // the one after
```

--------------------------------

### Parser usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/dql-parser.rst

This example shows how to instantiate the Parser with a DQL query and obtain the AST. The getAST method returns a SelectStatement AST object.

```php
<?php

$parser = new Parser('SELECT u FROM User u');
$AST = $parser->getAST(); // returns \Doctrine\ORM\Query\AST\SelectStatement
```

--------------------------------

### skipUntil() usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Example showing how to skip tokens until an operator token is reached.

```php
$lexer->setInput('price=10');
$lexer->moveNext();
$lexer->skipUntil('operator');
// lookahead is now the "=" token (type 'operator')
```

--------------------------------

### moveNext() usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Example showing the behavior of moveNext on a simple input, including return values and token transitions.

```php
$lexer->setInput('price=10'); // lookahead is null after setInput
$lexer->moveNext();           // true; token=null,    lookahead="price"
$lexer->moveNext();           // true; token="price", lookahead="="
$lexer->moveNext();           // true; token="=",     lookahead=10
$lexer->moveNext();           // false; token=10,     lookahead=null
```

--------------------------------

### isA() usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Example showing how to check if a value is of a specific token type.

```php
if ($lexer->isA('11', MyLexer::T_INTEGER)) {
    // getType('11') returned T_INTEGER
}
```

--------------------------------

### Install Doctrine Lexer via Composer

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

Install the library using Composer. Requires PHP ^8.1.

```bash
composer require doctrine/lexer
```

--------------------------------

### Creating a Token instance

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Example of instantiating a Token with a string value, type, and position. The comment shows the resulting property values.

```php
use Doctrine\Common\Lexer\Token;

$token = new Token('price', 'identifier', 0);
// $token->value === 'price', $token->type === 'identifier', $token->position === 0
```

--------------------------------

### resetPeek() method signature and usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Resets the peek pointer to 0, so the next peek() returns the token at the current position again. The example shows peeking twice, resetting, and peeking again to get the first token.

```php
/**
 * Resets the peek pointer to 0.
 *
 * @return void
 */
public function resetPeek()
```

```php
$lexer->peek();      // token A
$lexer->peek();      // token B (peek counter advanced)
$lexer->resetPeek();
$lexer->peek();      // token A again
```

--------------------------------

### isA() usage examples

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Examples of using isA() with a Token instance. Shows that it returns true when the token type matches any of the given types, and false otherwise. Note that a null type never matches.

```php
$token = new Token('=', 'operator', 5);

$token->isA('operator');          // true
$token->isA('int', 'operator');   // true
$token->isA('int');               // false
```

--------------------------------

### glimpse() usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Shows how to call glimpse() twice to retrieve the same token, as the peek counter is reset after each call.

```php
$token = $lexer->glimpse();  // token at current position
$again = $lexer->glimpse();  // same token, peek counter was reset
```

--------------------------------

### isNextTokenAny() usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Example usage of isNextTokenAny to check if the next token is a plus or minus.

```php
if ($lexer->isNextTokenAny([MyLexer::T_PLUS, MyLexer::T_MINUS])) {
    // next token is a plus or minus
}
```

--------------------------------

### getInputUntilPosition() method signature and usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Retrieves the original input up to but not including the character at the given position. The example shows setting input and getting the substring up to position 5.

```php
/**
 * Retrieve the original lexer's input until a given position.
 *
 * @return string
 */
public function getInputUntilPosition(int $position)
```

```php
$lexer->setInput('price=10');
$lexer->getInputUntilPosition(5); // "price"
```

--------------------------------

### setInput() method signature and usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Sets the input data to be tokenized, resetting the lexer and immediately tokenizing the new input. The example shows creating a lexer, setting input, moving to the next token, and accessing the lookahead value.

```php
/**
 * Sets the input data to be tokenized.
 *
 * The Lexer is immediately reset and the new input tokenized.
 * Any unprocessed tokens from any previous input are lost.
 *
 * @param string $input The input to be tokenized.
 *
 * @return void
 */
public function setInput(string $input)
```

```php
$lexer = new MyLexer();
$lexer->setInput('price=10');
$lexer->moveNext();
echo $lexer->lookahead->value; // "price"
```

--------------------------------

### Output of getUpperCaseCharacters example

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/simple-parser-example.rst

Shows the expected output when running the example: an array containing the uppercase characters B, E, H, and J.

```php
Array
(
    [0] => B
    [1] => E
    [2] => H
    [3] => J
)
```

--------------------------------

### reset() method signature and usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Resets the lexer state by setting lookahead, token, peek, and position to null or zero. It does not clear the token list or re-scan input. The example shows resetting and then moving to the next token.

```php
/**
 * Resets the lexer.
 *
 * @return void
 */
public function reset()
```

```php
$lexer->reset();          // back to initial state
$lexer->moveNext();       // re-reads tokens[0] as the new lookahead
```

--------------------------------

### isNextToken() method signature and usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Checks whether the current lookahead token matches the given type. The example shows setting input, moving to the next token, and checking if it is a string token.

```php
/**
 * Checks whether a given token matches the current lookahead.
 *
 * @param T $type
 *
 * @return bool
 *
 * @phpstan-assert-if-true !=null $this->lookahead
 */
public function isNextToken(int|string|UnitEnum $type)
```

```php
$lexer->setInput('price=10');
$lexer->moveNext();
if ($lexer->isNextToken(MyLexer::T_STRING)) {
    // lookahead is guaranteed non-null here
}
```

--------------------------------

### isA() with AbstractLexer token

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Practical example of using isA() on a token obtained from an AbstractLexer, checking if the token type is T_UPPER and collecting its value.

```php
$lexer->moveNext();
if ($lexer->token !== null && $lexer->token->isA(CharacterTypeLexer::T_UPPER)) {
    $upper[] = $lexer->token->value;
}
```

--------------------------------

### resetPosition() method signature and usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Resets the lexer position to the given character offset, positioning the cursor at the first token whose position is at or after the offset. The example shows setting input, moving through tokens, resetting to position 5, and moving again.

```php
/**
 * Resets the lexer position on the input to the given position.
 *
 * @param int $position Character offset into the input, as found in
 *                      Token::$position.
 *
 * @return void
 */
public function resetPosition(int $position = 0)
```

```php
$lexer->setInput('price=10');
$lexer->moveNext(); // lookahead: "price" (position 0)
$lexer->moveNext(); // lookahead: "=" (position 5)
$lexer->resetPosition(5);
$lexer->moveNext(); // lookahead: "=" again
```

--------------------------------

### getLiteral() usage example

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Demonstrates getLiteral() with different token types: a class constant, a string, and a UnitEnum case. Returns the class name with constant name for known tokens, or the token value itself if not found.

```php
echo $lexer->getLiteral(MyLexer::T_STRING);  // "App\MyLexer::T_STRING"
echo $lexer->getLiteral('fake_token');        // "fake_token"
echo $lexer->getLiteral(TokenType::INT);      // "App\TokenType::INT"
```

--------------------------------

### getLiteral()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Gets the literal name for a given token type, returning a class constant name or the token value itself if no match is found.

```APIDOC
## getLiteral()

### Description
Gets the literal for a given token. If the token is a `UnitEnum` instance, returns `TokenType::CASE_NAME`. Otherwise, reflects over the constants of the concrete lexer class and returns `ClassName::CONSTANT_NAME` on a strict value match. If no matching constant is found, the token value itself is returned unchanged.

### Method
`getLiteral()`

### Signature
```php
public function getLiteral(int|string|UnitEnum $token): int|string
```

### Parameters
- **$token** (`T` = `int|string|UnitEnum`) - Required - The token type to name.

### Return Value
- **Type:** `int|string`
- **Description:** The literal name of the token, either as a class constant reference or the original token value.

### Example
```php
echo $lexer->getLiteral(MyLexer::T_STRING);  // "App\MyLexer::T_STRING"
echo $lexer->getLiteral('fake_token');        // "fake_token"
echo $lexer->getLiteral(TokenType::INT);      // "App\TokenType::INT"
```
```

--------------------------------

### Use CharacterTypeLexer to extract uppercase characters

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/simple-parser-example.rst

Defines UpperCaseCharacterExtracter which uses CharacterTypeLexer to iterate over tokens and collect uppercase characters. Includes an example usage that prints the result.

```php
<?php

class UpperCaseCharacterExtracter
{
    public function __construct(private CharacterTypeLexer $lexer)
    {
    }

    /** @return list<string> */
    public function getUpperCaseCharacters(string $string): array
    {
        $this->lexer->setInput($string);
        $this->lexer->moveNext();

        $upperCaseChars = [];
        while (true) {
            if (!$this->lexer->lookahead) {
                break;
            }

            $this->lexer->moveNext();

            if ($this->lexer->token->isA(CharacterTypeLexer::T_UPPER)) {
                $upperCaseChars[] = $this->lexer->token->value;
            }
        }

        return $upperCaseChars;
    }
}

$upperCaseCharacterExtractor = new UpperCaseCharacterExtracter(new CharacterTypeLexer());
$upperCaseCharacters = $upperCaseCharacterExtractor->getUpperCaseCharacters('1aBcdEfgHiJ12');

print_r($upperCaseCharacters);
```

--------------------------------

### Iterate tokens with AbstractLexer

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Shows how to use setInput() and moveNext() to walk through tokens, accessing the current token via $lexer->token. The output demonstrates the token types, values, and positions.

```php
$lexer->setInput('price=10');

while ($lexer->moveNext()) {
    $token = $lexer->token; // Token<'string'|'operator'|'int', string|int>
    printf(
        "type=%s value=%s position=%d\n",
        (string) $token->type,
        (string) $token->value,
        $token->position,
    );
}
// type=string value=price position=0
// type=operator value== position=5
// type=int value=10 position=6
```

--------------------------------

### Import Doctrine Lexer classes

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

Import the AbstractLexer and Token classes for use in your code.

```php
use Doctrine\Common\Lexer\AbstractLexer;
use Doctrine\Common\Lexer\Token;
```

--------------------------------

### Import Token class

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Import statement for the Token class.

```php
use Doctrine\Common\Lexer\Token;
```

--------------------------------

### Extending AbstractLexer in PHP

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Shows the import path and a minimal subclass of AbstractLexer. The subclass must implement getCatchablePatterns(), getNonCatchablePatterns(), and getType().

```php
use Doctrine\Common\Lexer\AbstractLexer;

/** @extends AbstractLexer<string, string|int> */
final class MyLexer extends AbstractLexer
{
    // implement getCatchablePatterns(), getNonCatchablePatterns(), getType()
}
```

--------------------------------

### Token constructor signature

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Shows the constructor signature with parameters for value, type, and position. The type parameter has no native type declaration; semantics come from the docblock.

```php
/**
 * @param V      $value
 * @param T|null $type
 */
public function __construct(string|int|float|bool $value, $type, int $position)
```

--------------------------------

### glimpse()

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Peeks at the next token, returns it and immediately resets the peek.

```APIDOC
## glimpse()

### Description
Peeks at the next token, returns it and immediately resets the peek.

### Method
glimpse

### Parameters
None.

### Response
Returns the next token.
```

--------------------------------

### Parser class with Lexer usage

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/dql-parser.rst

This is a rough implementation of the DQL Parser that uses the Lexer. It shows the constructor, getAST method, and QueryLanguage method that parses DQL into an AST. The full implementation is available at the provided GitHub link.

```php
<?php

class Parser
{
    private $lexer;

    public function __construct(string $dql)
    {
        $this->lexer = new Lexer($dql);
    }

    // ...

    public function getAST(): AST\SelectStatement|AST\UpdateStatement|AST\DeleteStatement
    {
        // Parse & build AST
        $AST = $this->QueryLanguage();

        // ...

        return $AST;
    }

    public function QueryLanguage(): AST\SelectStatement|AST\UpdateStatement|AST\DeleteStatement
    {
        $this->lexer->moveNext();

        switch ($this->lexer->lookahead->type) {
            case Lexer::T_SELECT:
                $statement = $this->SelectStatement();
                break;
            case Lexer::T_UPDATE:
                $statement = $this->UpdateStatement();
                break;
            case Lexer::T_DELETE:
                $statement = $this->DeleteStatement();
                break;
            default:
                $this->syntaxError('SELECT, UPDATE or DELETE');
        }

        // Check for end of string
        if ($this->lexer->lookahead !== null) {
            $this->syntaxError('end of string');
        }

        return $statement;
    }

    // ...
}
```

--------------------------------

### Token::__construct

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Creates a new Token instance with the given value, type, and position. The constructor performs no validation.

```APIDOC
## Token::__construct

### Description
Creates a new immutable Token value object representing a single scanned token.

### Method
`public function __construct(string|int|float|bool $value, $type, int $position)`

### Parameters
- `$value` (string|int|float|bool) - Required - The string value of the token in the input string; may already be transformed (e.g. '10' → int 10) by the lexer's getType().
- `$type` (UnitEnum|string|int|null) - Required - The token type (identifier, numeric, string, keyword, …) or null. Parameter has no native type declaration; semantics come from the docblock.
- `$position` (int) - Required - The character offset of the token in the input string.

### Return
A new Token instance. No @throws annotation — the constructor performs no validation.

### Example
```php
use Doctrine\Common\Lexer\Token;

$token = new Token('price', 'identifier', 0);
// $token->value === 'price', $token->type === 'identifier', $token->position === 0
```
```

--------------------------------

### peek() method signature

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Method signature for peek, which moves the lookahead token forward without consuming it.

```php
/**
 * Moves the lookahead token forward.
 *
 * @return Token<T, V>|null The next token or NULL if there are no more tokens ahead.
 */
public function peek()
```

--------------------------------

### Doctrine ORM DQL Lexer implementation

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/dql-parser.rst

A complete PHP class extending AbstractLexer for tokenizing DQL queries. It defines token constants for operators, identifiers, and keywords, and implements getCatchablePatterns, getNonCatchablePatterns, and getType methods. The constructor accepts a query string and calls setInput.

```php
<?php

use Doctrine\Common\Lexer\AbstractLexer;

class Lexer extends AbstractLexer
{
    // All tokens that are not valid identifiers must be < 100
    public const T_NONE              = 1;
    public const T_INTEGER           = 2;
    public const T_STRING            = 3;
    public const T_INPUT_PARAMETER   = 4;
    public const T_FLOAT             = 5;
    public const T_CLOSE_PARENTHESIS = 6;
    public const T_OPEN_PARENTHESIS  = 7;
    public const T_COMMA             = 8;
    public const T_DIVIDE            = 9;
    public const T_DOT               = 10;
    public const T_EQUALS            = 11;
    public const T_GREATER_THAN      = 12;
    public const T_LOWER_THAN        = 13;
    public const T_MINUS             = 14;
    public const T_MULTIPLY          = 15;
    public const T_NEGATE            = 16;
    public const T_PLUS              = 17;
    public const T_OPEN_CURLY_BRACE  = 18;
    public const T_CLOSE_CURLY_BRACE = 19;

    // All tokens that are identifiers or keywords that could be considered as identifiers should be >= 100
    public const T_ALIASED_NAME         = 100;
    public const T_FULLY_QUALIFIED_NAME = 101;
    public const T_IDENTIFIER           = 102;

    // All keyword tokens should be >= 200
    public const T_ALL      = 200;
    public const T_AND      = 201;
    public const T_ANY      = 202;
    public const T_AS       = 203;
    public const T_ASC      = 204;
    public const T_AVG      = 205;
    public const T_BETWEEN  = 206;
    public const T_BOTH     = 207;
    public const T_BY       = 208;
    public const T_CASE     = 209;
    public const T_COALESCE = 210;
    public const T_COUNT    = 211;
    public const T_DELETE   = 212;
    public const T_DESC     = 213;
    public const T_DISTINCT = 214;
    public const T_ELSE     = 215;
    public const T_EMPTY    = 216;
    public const T_END      = 217;
    public const T_ESCAPE   = 218;
    public const T_EXISTS   = 219;
    public const T_FALSE    = 220;
    public const T_FROM     = 221;
    public const T_GROUP    = 222;
    public const T_HAVING   = 223;
    public const T_HIDDEN   = 224;
    public const T_IN       = 225;
    public const T_INDEX    = 226;
    public const T_INNER    = 227;
    public const T_INSTANCE = 228;
    public const T_IS       = 229;
    public const T_JOIN     = 230;
    public const T_LEADING  = 231;
    public const T_LEFT     = 232;
    public const T_LIKE     = 233;
    public const T_MAX      = 234;
    public const T_MEMBER   = 235;
    public const T_MIN      = 236;
    public const T_NEW      = 237;
    public const T_NOT      = 238;
    public const T_NULL     = 239;
    public const T_NULLIF   = 240;
    public const T_OF       = 241;
    public const T_OR       = 242;
    public const T_ORDER    = 243;
    public const T_OUTER    = 244;
    public const T_PARTIAL  = 245;
    public const T_SELECT   = 246;
    public const T_SET      = 247;
    public const T_SOME     = 248;
    public const T_SUM      = 249;
    public const T_THEN     = 250;
    public const T_TRAILING = 251;
    public const T_TRUE     = 252;
    public const T_UPDATE   = 253;
    public const T_WHEN     = 254;
    public const T_WHERE    = 255;
    public const T_WITH     = 256;

    /**
     * Creates a new query scanner object.
     *
     * @param string $input A query string.
     */
    public function __construct(string $input)
    {
        $this->setInput($input);
    }

    /**
     * {@inheritdoc}
     */
    protected function getCatchablePatterns(): array
    {
        return [
            '[a-z_][a-z0-9_]*\:[a-z_][a-z0-9_]*(?:\\\[a-z_][a-z0-9_]*)*', // aliased name
            '[a-z_\\\][a-z0-9_]*(?:\\\[a-z_][a-z0-9_]*)*', // identifier or qualified name
            '(?:[0-9]+(?:[\.][0-9]+)*)(?:e[+-]?[0-9]+)?', // numbers
            "'(?:[^']|'')*'", // quoted strings
            '\?[0-9]*|:[a-z_][a-z0-9_]*', // parameters
        ];
    }

    /**
     * {@inheritdoc}
     */
    protected function getNonCatchablePatterns(): array
    {
        return ['\s+', '(.)'];
    }

    /**
     * {@inheritdoc}
     */
    protected function getType(&$value): int
    {
        $type = self::T_NONE;

```

--------------------------------

### peek()

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Moves the lookahead token forward.

```APIDOC
## peek()

### Description
Moves the lookahead token forward.

### Method
peek

### Parameters
None.

### Response
No return value specified.
```

--------------------------------

### Define and use a custom AbstractLexer subclass to collect upper-case characters

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

Shows how to subclass Doctrine\Common\Lexer\AbstractLexer, implementing getCatchablePatterns(), getNonCatchablePatterns(), and getType() to classify single characters. The lexer is then fed '1aBcdEfgHiJ12' and iterated to collect upper-case tokens, resulting in $upper === ['B', 'E', 'H', 'J']. The token type constants T_UPPER, T_LOWER, and T_NUMBER are defined in the subclass.

```php
<?php

use Doctrine\Common\Lexer\AbstractLexer;

/**
 * @extends AbstractLexer<self::T_*, string>
 */
final class CharacterTypeLexer extends AbstractLexer
{
    public const T_UPPER  = 1;
    public const T_LOWER  = 2;
    public const T_NUMBER = 3;

    protected function getCatchablePatterns(): array
    {
        return ['[a-zA-Z0-9]'];
    }

    protected function getNonCatchablePatterns(): array
    {
        return [];
    }

    protected function getType(string &$value): int
    {
        if (is_numeric($value)) return self::T_NUMBER;
        if (strtoupper($value) === $value) return self::T_UPPER;
        return self::T_LOWER;
    }
}

$lexer = new CharacterTypeLexer();
$lexer->setInput('1aBcdEfgHiJ12');
$lexer->moveNext();

$upper = [];
while (true) {
    if ($lexer->lookahead === null) break;
    $lexer->moveNext();
    if ($lexer->token->isA(CharacterTypeLexer::T_UPPER)) {
        $upper[] = $lexer->token->value;
    }
}

// $upper === ['B', 'E', 'H', 'J']
```

--------------------------------

### moveNext()

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Moves to the next token in the input string.

```APIDOC
## moveNext()

### Description
Moves to the next token in the input string.

### Method
moveNext

### Parameters
None.

### Response
No return value specified.
```

--------------------------------

### isA($value, $token)

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Checks if given value is identical to the given token.

```APIDOC
## isA($value, $token)

### Description
Checks if given value is identical to the given token.

### Method
isA

### Parameters
- **$value** (mixed) - Required - The value to check.
- **$token** (mixed) - Required - The token to compare against.

### Response
Returns a boolean indicating whether the value is identical to the token.
```

--------------------------------

### Abstract methods to implement in AbstractLexer

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

These are the three abstract methods that must be implemented when extending Doctrine\Common\Lexer\AbstractLexer. They define the lexical catchable and non-catchable patterns and a method to retrieve the token type while optionally processing the token value.

```php
/**
 * Lexical catchable patterns.
 *
 * @return string[]
 */
abstract protected function getCatchablePatterns();

/**
 * Lexical non-catchable patterns.
 *
 * @return string[]
 */
abstract protected function getNonCatchablePatterns();

/** Retrieve token type. Also processes the token value if necessary. */
abstract protected function getType(string &$value): int;
```

--------------------------------

### Composer PSR-4 autoload configuration

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

Autoload configuration mapping the Doctrine\Common\Lexer namespace to the src directory.

```json
"autoload": {
    "psr-4": {
        "Doctrine\\Common\\Lexer\\": "src"
    }
}
```

--------------------------------

### Module graph of Doctrine\Common\Lexer

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

ASCII diagram showing the relationships between AbstractLexer and Token classes.

```text
Doctrine\Common\Lexer
├── AbstractLexer  (abstract class, src/AbstractLexer.php)
│   ├── uses Token (lookahead/token properties, scan() output)
│   ├── uses ReflectionClass  (getLiteral())
│   ├── uses preg_split / PREG_SPLIT_* constants  (scan())
│   └── requires subclass to implement 3 abstract methods
└── Token  (final class, src/Token.php)
    └── uses UnitEnum  (type comparison in isA())
```

--------------------------------

### moveNext() method signature

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Method signature for moveNext, which moves to the next token in the input string. Includes phpstan annotations.

```php
/**
 * Moves to the next token in the input string.
 *
 * @return bool
 *
 * @phpstan-impure
 * @phpstan-assert-if-true !null $this->lookahead
 */
public function moveNext()
```

--------------------------------

### setInput($input)

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Sets the input data to be tokenized. The Lexer is immediately reset and the new input tokenized.

```APIDOC
## setInput($input)

### Description
Sets the input data to be tokenized. The Lexer is immediately reset and the new input tokenized.

### Method
setInput

### Parameters
- **$input** (string) - Required - The input data to be tokenized.

### Response
No return value specified.
```

--------------------------------

### Public properties $lookahead and $token

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Declares the public properties for the lexer cursor. $lookahead holds the next token (initially null), and $token holds the last matched token (initially null). Both are nullable Token instances.

```php
/**
 * The next token in the input.
 *
 * @var Token<T, V>|null
 */
public Token|null $lookahead;

/**
 * The last matched/seen token.
 *
 * @var Token<T, V>|null
 */
public Token|null $token;

```

--------------------------------

### resetPeek()

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Resets the peek pointer to 0.

```APIDOC
## resetPeek()

### Description
Resets the peek pointer to 0.

### Method
resetPeek

### Parameters
None.

### Response
No return value specified.
```

--------------------------------

### skipUntil() method signature

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Method signature for skipUntil, which skips input tokens until it sees a token with the given value.

```php
/**
 * Tells the lexer to skip input tokens until it sees a token with the given value.
 *
 * @param T $type The token type to skip until.
 *
 * @return void
 */
public function skipUntil(int|string|UnitEnum $type)
```

--------------------------------

### skipUntil()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Skips input tokens until it sees a token with the given type. Repeatedly calls moveNext() while the current lookahead is non-null and not of the specified type.

```APIDOC
## skipUntil()

### Description
Skips input tokens until it sees a token with the given type. Repeatedly calls moveNext() while the current lookahead is non-null and not of the specified type.

### Method
skipUntil

### Parameters
- `$type` (T) - Required - The token type to skip until.

### Return
`void`

### Behavior
Repeatedly calls `moveNext()` while the current lookahead is non-`null` and is **not** of `$type`. If the current lookahead already matches, nothing is consumed; if the end of input is reached first, it stops with `$lookahead === null`.

### Example
```php
$lexer->setInput('price=10');
$lexer->moveNext();
$lexer->skipUntil('operator');
// lookahead is now the "=" token (type 'operator')
```
```

--------------------------------

### moveNext()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Moves to the next token in the input string. Returns true if a new lookahead token is available, false at the end of tokens.

```APIDOC
## moveNext()

### Description
Moves to the next token in the input string. Returns true if a new lookahead token is available, false at the end of tokens.

### Method
moveNext

### Parameters
No parameters.

### Return
`bool` — `true` if there is a new `$this->lookahead` (non-`null`), `false` at end of tokens.

### Behavior
Sets `$this->peek = 0`, moves the current `$lookahead` into `$token`, then advances: `$lookahead = $this->tokens[$this->position++]` (or `null` past the end).

### Example
```php
$lexer->setInput('price=10'); // lookahead is null after setInput
$lexer->moveNext();           // true; token=null,    lookahead="price"
$lexer->moveNext();           // true; token="price", lookahead="="
$lexer->moveNext();           // true; token="=",     lookahead=10
$lexer->moveNext();           // false; token=10,     lookahead=null
```
```

--------------------------------

### isA()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Checks if a given value is identical to the given token type by comparing the classified value to the token type.

```APIDOC
## isA()

### Description
Checks if a given value is identical to the given token type by comparing the classified value to the token type.

### Method
isA

### Parameters
- `$value` (string) - Required - A value to classify.
- `$token` (T) - Required - The token type to compare against.

### Return
`bool` — `$this->getType($value) === $token`.

### Example
```php
if ($lexer->isA('11', MyLexer::T_INTEGER)) {
    // getType('11') returned T_INTEGER
}
```
```

--------------------------------

### reset()

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Resets the lexer.

```APIDOC
## reset()

### Description
Resets the lexer.

### Method
reset

### Parameters
None.

### Response
No return value specified.
```

--------------------------------

### Token class public property declarations

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Shows the public property declarations for the Token class. All properties are readonly and assigned once in the constructor.

```php
public string|int|float|bool $value;
public $type;
public int $position;
```

--------------------------------

### isA() method signature

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Signature of the isA() method, which accepts variadic token types and returns a boolean. The method uses strict comparison to check if the token's type matches any of the given types.

```php
/** @param T ...$types */
public function isA(...$types): bool
```

--------------------------------

### getInputUntilPosition()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Retrieves the original lexer's input until a given character position.

```APIDOC
## getInputUntilPosition()

### Description
Retrieve the original lexer's input until a given position. Returns `substr($this->input, 0, $position)`, i.e. the original input up to but not including the character at `$position`.

### Method
getInputUntilPosition

### Signature
```php
public function getInputUntilPosition(int $position)
```

### Parameters
- `$position` (int) - Required - Character offset into the original input.

### Return
`string` — `substr($this->input, 0, $position)`.

### Example
```php
$lexer->setInput('price=10');
$lexer->getInputUntilPosition(5); // "price"
```
```

--------------------------------

### isA()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Checks if the token's type strictly equals any of the given types. Returns true if a match is found, false otherwise.

```APIDOC
## isA()

### Description
Checks if the token's type strictly equals any of the given types. Returns true if a match is found, false otherwise.

### Method
isA

### Signature
```php
public function isA(...$types): bool
```

### Parameters
- `$types` (variadic `UnitEnum|string|int`) - Required - One or more token types to test against `$this->type`.

### Return Value
- `bool` - `true` if `$this->type` strictly equals any of the given types, `false` otherwise.

### Example
```php
$token = new Token('=', 'operator', 5);

$token->isA('operator');          // true
$token->isA('int', 'operator');   // true
$token->isA('int');               // false
```
```

--------------------------------

### Token class declaration with template annotations

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/token.md

Shows the final class declaration with template parameters T and V, including variance annotations. The class is final and cannot be subclassed.

```php
/**
 * @template T of UnitEnum|string|int
 * @template-covariant V of string|int|float|bool
 */
final class Token
```

--------------------------------

### skipUntil($type)

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Tells the lexer to skip input tokens until it sees a token with the given value.

```APIDOC
## skipUntil($type)

### Description
Tells the lexer to skip input tokens until it sees a token with the given value.

### Method
skipUntil

### Parameters
- **$type** (mixed) - Required - The token value to skip until.

### Response
No return value specified.
```

--------------------------------

### AbstractLexer Class

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

Abstract base class for implementing custom lexers. Provides methods to set input, iterate over tokens, and inspect token types.

```APIDOC
## AbstractLexer<T of UnitEnum|string|int, V of string|int|float|bool>

### Description
Abstract base class for building a lexer. Subclasses must implement the `getType()` method to map token values to types.

### Public Properties
- **$lookahead** (Token|null) - The next token to be returned by `moveNext()`.
- **$token** (Token|null) - The current token.

### Public Methods

#### setInput()
Sets the input string to be tokenized.

#### reset()
Resets the lexer to its initial state.

#### resetPeek()
Resets the peek pointer to the current token.

#### resetPosition()
Resets the position to the beginning of the input.

#### getInputUntilPosition()
Returns the input string from the current position up to the given position.

#### isNextToken()
Checks if the next token matches the given type.

#### isNextTokenAny()
Checks if the next token matches any of the given types.

#### moveNext()
Moves to the next token and returns true if a token is available, false otherwise.

#### skipUntil()
Skips tokens until a token of the given type is found.

#### isA()
Checks if the given token is of the given type.

#### peek()
Returns the next token without advancing the lexer.

#### glimpse()
Returns the token after the next token without advancing the lexer.

#### getLiteral()
Returns the literal value of a token type.
```

--------------------------------

### isNextTokenAny() method signature

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Method signature for isNextTokenAny, which checks if any of the given token types matches the current lookahead. Includes phpstan assertions.

```php
/**
 * Checks whether any of the given tokens matches the current lookahead.
 *
 * @param list<T> $types
 *
 * @return bool
 *
 * @phpstan-assert-if-true !=null $this->lookahead
 */
public function isNextTokenAny(array $types)
```

--------------------------------

### isNextToken($token)

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Checks whether a given token matches the current lookahead.

```APIDOC
## isNextToken($token)

### Description
Checks whether a given token matches the current lookahead.

### Method
isNextToken

### Parameters
- **$token** (mixed) - Required - The token to check against the current lookahead.

### Response
Returns a boolean indicating whether the token matches.
```

--------------------------------

### resetPosition()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Resets the lexer position on the input to the given character offset. Positions the cursor at the first token whose position is at or after the given offset.

```APIDOC
## resetPosition()

### Description
Resets the lexer position on the input to the given position. Positions the cursor at the first token whose `Token::$position` is **at or after** the given character offset (a linear scan over the internal token list, `findTokenIndexAtOrAfterOffset()`, `src/AbstractLexer.php:132`). If no token starts at or after the offset, the cursor lands one past the end and the next `moveNext()` returns `false`. This is a character offset, **not** a token index.

### Method
resetPosition

### Signature
```php
public function resetPosition(int $position = 0)
```

### Parameters
- `$position` (int) - Optional - Default: `0` - Character offset into the input, as found in `Token::$position`.

### Return
`void`.

### Example
```php
$lexer->setInput('price=10');
$lexer->moveNext(); // lookahead: "price" (position 0)
$lexer->moveNext(); // lookahead: "=" (position 5)
$lexer->resetPosition(5);
$lexer->moveNext(); // lookahead: "=" again
```
```

--------------------------------

### Regex composition in scan()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

The scan() method composes a single regex from the catchable and non-catchable patterns. Catchable patterns are wrapped in capturing groups, non-catchable are not. Default modifiers are 'iu'.

```php
sprintf('/(%s)|%s/%s',
    implode(')|(', $this->getCatchablePatterns()),
    implode('|', $this->getNonCatchablePatterns()),
    $this->getModifiers(),
);
```

--------------------------------

### isNextTokenAny(array $tokens)

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Checks whether any of the given tokens matches the current lookahead.

```APIDOC
## isNextTokenAny(array $tokens)

### Description
Checks whether any of the given tokens matches the current lookahead.

### Method
isNextTokenAny

### Parameters
- **$tokens** (array) - Required - An array of tokens to check against the current lookahead.

### Response
Returns a boolean indicating whether any token matches.
```

--------------------------------

### resetPosition($position = 0)

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/index.rst

Resets the lexer position on the input to the given position.

```APIDOC
## resetPosition($position = 0)

### Description
Resets the lexer position on the input to the given position.

### Method
resetPosition

### Parameters
- **$position** (int) - Optional - The position to reset to. Defaults to 0.

### Response
No return value specified.
```

--------------------------------

### Define CharacterTypeLexer extending AbstractLexer

Source: https://github.com/doctrine/lexer/blob/3.0.x/docs/en/simple-parser-example.rst

Extends Doctrine\Common\Lexer\AbstractLexer to tokenize a string into T_UPPER, T_LOWER, and T_NUMBER tokens. Implements getCatchablePatterns, getNonCatchablePatterns, and getType methods.

```php
<?php

use Doctrine\Common\Lexer\AbstractLexer;

/**
 * @extends AbstractLexer<CharacterTypeLexer::T_*, string>
 */
class CharacterTypeLexer extends AbstractLexer
{
    const T_UPPER =  1;
    const T_LOWER =  2;
    const T_NUMBER = 3;

    protected function getCatchablePatterns(): array
    {
        return [
            '[a-zA-Z0-9]',
        ];
    }

    protected function getNonCatchablePatterns(): array
    {
        return [];
    }

    protected function getType(&$value): int
    {
        if (is_numeric($value)) {
            return self::T_NUMBER;
        }

        if (strtoupper($value) === $value) {
            return self::T_UPPER;
        }

        if (strtolower($value) === $value) {
            return self::T_LOWER;
        }
    }
}
```

--------------------------------

### reset()

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Resets the lexer to its initial state, clearing the lookahead, token, peek counter, and position. It does not clear the internal token list or re-scan the input.

```APIDOC
## reset()

### Description
Resets the lexer. Sets `$lookahead = null`, `$token = null`, `$peek = 0`, `$position = 0`. It does **not** clear the internal token list and does **not** re-scan the input; call `setInput()` to re-tokenize.

### Method
reset

### Signature
```php
public function reset()
```

### Parameters
None.

### Return
`void`.

### Example
```php
$lexer->reset();          // back to initial state
$lexer->moveNext();       // re-reads tokens[0] as the new lookahead
```
```

--------------------------------

### Token Class

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/index.md

Final value object representing a single token with its value, type, and position in the input.

```APIDOC
## Token<T of UnitEnum|string|int, V of string|int|float|bool>

### Description
Represents a single token produced by the lexer. Contains the token value, type, and position.

### Public Properties
- **$value** (V) - The value of the token.
- **$type** (T|null) - The type of the token.
- **$position** (int) - The position of the token in the input.

### Constructor
#### __construct()
Creates a new Token instance with the given value, type, and position.

### Public Methods
#### isA()
Checks if the token's type matches the given type.
```

--------------------------------

### AbstractLexer class declaration with template parameters

Source: https://github.com/doctrine/lexer/blob/3.0.x/_autodocs/api-reference/abstract-lexer.md

Declares the abstract base class for lexers. Template T is the token type (UnitEnum, string, or int), and V is the token value type (string, int, float, or bool). The class has no constructor and is not instantiable.

```php
/**
 * Base class for writing simple lexers, i.e. for creating small DSLs.
 *
 * @template T of UnitEnum|string|int
 * @template V of string|int|float|bool
 */
abstract class AbstractLexer

```
Tokens:
Raw