Tokenizer

https://github.com/theseer/tokenizer

Admin

A small PHP library for converting tokenized PHP source code into XML format.

Tokens:702

Snippets:3

Trust Score:9.2

License:BSD-3-Clause

Update:1 year ago

Show doc for...

Tokens:

Raw

### Install TheSeer Tokenizer with Composer

Source: https://github.com/theseer/tokenizer/blob/master/README.md

Demonstrates how to add the TheSeer Tokenizer library to a project using Composer, both as a regular dependency and a development-time dependency.

```bash
composer require theseer/tokenizer
```

```bash
composer require --dev theseer/tokenizer
```

--------------------------------

### Parse PHP File and Serialize to XML

Source: https://github.com/theseer/tokenizer/blob/master/README.md

Shows a practical example of using the TheSeer Tokenizer library. It involves creating a Tokenizer instance, parsing a PHP file's content, and then using an XMLSerializer to convert the tokens into an XML string.

```php
$tokenizer = new TheSeer\Tokenizer\Tokenizer();
$tokens = $tokenizer->parse(file_get_contents(__DIR__ . '/src/XMLSerializer.php'));

$serializer = new TheSeer\Tokenizer\XMLSerializer();
$xml = $serializer->toXML($tokens);

echo $xml;
```

--------------------------------

### Example XML Output Structure

Source: https://github.com/theseer/tokenizer/blob/master/README.md

Illustrates the expected XML structure generated by the TheSeer Tokenizer library after processing PHP source code. It shows how tokens are represented with their names and values within line elements.

```xml
<?xml version="1.0"?>
<source xmlns="https://github.com/theseer/tokenizer">
 <line no="1">
  <token name="T_OPEN_TAG">&lt;?php </token>
  <token name="T_DECLARE">declare</token>
  <token name="T_OPEN_BRACKET">(</token>
  <token name="T_STRING">strict_types</token>
  <token name="T_WHITESPACE"> </token>
  <token name="T_EQUAL">=</token>
  <token name="T_WHITESPACE"> </token>
  <token name="T_LNUMBER">1</token>
  <token name="T_CLOSE_BRACKET">)</token>
  <token name="T_SEMICOLON">;</token>
 </line>
</source>
```

Tokens:

Raw