### Parsing Configuration-like Syntax with Nested Structures in C#

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

This C# example showcases advanced usage of TinyTokenizer by parsing a configuration-like string with nested blocks. It defines a recursive function `ProcessTokens` to traverse the token tree and print the hierarchical structure, demonstrating how to handle nested declarations and different token types within a configuration context.

```csharp
using TinyTokenizer;
using System.Collections.Immutable;

var config = @"
server {
    host: localhost
    port: 8080
    routes [
        /api, /web, /admin
    ]
}
";

var tokenizer = new Tokenizer(config.AsMemory());
var tokens = tokenizer.Tokenize();

void ProcessTokens(ImmutableArray<Token> tokens, int indent = 0)
{
    var prefix = new string(' ', indent * 2);

    foreach (var token in tokens)
    {
        switch (token)
        {
            case TextToken text:
                Console.WriteLine($"{prefix}Text: {text.ContentSpan.ToString()}");
                break;

            case BlockToken block:
                Console.WriteLine($"{prefix}Block {block.OpeningDelimiter}{block.ClosingDelimiter}:");
                ProcessTokens(block.Children, indent + 1);
                break;

            case SymbolToken symbol:
                Console.WriteLine($"{prefix}Symbol: {symbol.Symbol}");
                break;
        }
    }
}

ProcessTokens(tokens);

// Output shows hierarchical structure:
// Text: server
// Block {}:
//   Text: host
//   Symbol: :
//   Text: localhost
//   Text: port
//   Symbol: :
//   Text: 8080
//   Text: routes
//   Block []:
//     Text: /api
//     Symbol: ,
//     Text: /web
//     Symbol: ,
//     Text: /admin
```

--------------------------------

### Parse Nested Blocks Recursively

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Shows the recursive parsing capabilities of TinyTokenizer for deeply nested blocks. The example demonstrates navigating through multiple levels of block tokens (e.g., {}, [], ()).

```csharp
using TinyTokenizer;

var source = "{outer [inner (deepest)]}";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

// Navigate the nested structure
var braceBlock = (BlockToken)tokens[0];
Console.WriteLine($"Level 1: {braceBlock.FullContent.ToString()}");

// braceBlock.Children contains: TextToken("outer"), WhitespaceToken(" "), BlockToken("[inner (deepest)]")
var bracketBlock = (BlockToken)braceBlock.Children[2];
Console.WriteLine($"Level 2: {bracketBlock.FullContent.ToString()}");

// bracketBlock.Children contains: TextToken("inner"), WhitespaceToken(" "), BlockToken("(deepest)")
var parenBlock = (BlockToken)bracketBlock.Children[2];
Console.WriteLine($"Level 3: {parenBlock.FullContent.ToString()}");
Console.WriteLine($"Deepest content: {parenBlock.InnerContent.ToString()}");

// Output:
// Level 1: {outer [inner (deepest)]}
// Level 2: [inner (deepest)]
// Level 3: (deepest)
// Deepest content: deepest
```

--------------------------------

### Creating a Tokenizer and Parsing Text

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Demonstrates basic tokenization of a string into an immutable array of tokens using default options.

```APIDOC
## Creating a Tokenizer and Parsing Text

### Description
Basic tokenization of a string into an immutable array of tokens using default options.

### Method
POST (Conceptual - this is a library usage example, not a REST endpoint)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```csharp
using TinyTokenizer;

// Parse a simple function call
var source = "func(a, b)";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

// Result: 2 tokens
// tokens[0]: TextToken("func")
// tokens[1]: BlockToken("(a, b)") with 4 children:
//   - TextToken("a")
//   - SymbolToken(",")
//   - WhitespaceToken(" ")
//   - TextToken("b")

Console.WriteLine($"Found {tokens.Length} top-level tokens");
```

### Response
#### Success Response (200)
- **tokens** (Array<IToken>) - An array of top-level tokens representing the parsed text.

#### Response Example
```json
{
  "tokens": [
    {
      "type": "TextToken",
      "content": "func"
    },
    {
      "type": "BlockToken",
      "content": "(a, b)",
      "openingDelimiter": "(",
      "closingDelimiter": ")",
      "children": [
        {
          "type": "TextToken",
          "content": "a"
        },
        {
          "type": "SymbolToken",
          "content": ","
        },
        {
          "type": "WhitespaceToken",
          "content": " "
        },
        {
          "type": "TextToken",
          "content": "b"
        }
      ]
    }
  ]
}
```
```

--------------------------------

### Working with BlockToken Properties

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Explains how to access the content, structure, delimiters, and children of `BlockToken` instances.

```APIDOC
## Working with BlockToken Properties

### Description
Accessing the content and structure of declaration blocks including delimiters and children.

### Method
POST (Conceptual - this is a library usage example, not a REST endpoint)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```csharp
using TinyTokenizer;

var source = "{inner content}";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

var block = (BlockToken)tokens[0];

// Full content includes delimiters
Console.WriteLine(block.FullContent.ToString());      // "{inner content}"

// Inner content excludes delimiters
Console.WriteLine(block.InnerContent.ToString());     // "inner content"

// Access delimiter characters
Console.WriteLine(block.OpeningDelimiter);            // '{'
Console.WriteLine(block.ClosingDelimiter);            // '}'

// Token type identifies the block type
Console.WriteLine(block.Type);                        // TokenType.BraceBlock

// Children are recursively parsed tokens
foreach (var child in block.Children)
{
    Console.WriteLine($"{child.GetType().Name}: {child.ContentSpan.ToString()}");
}
// Output: TextToken: inner
//         WhitespaceToken: 
//         TextToken: content
```

### Response
#### Success Response (200)
- **block.FullContent** (ReadOnlySpan<char>) - The complete content of the block, including delimiters.
- **block.InnerContent** (ReadOnlySpan<char>) - The content within the block, excluding delimiters.
- **block.OpeningDelimiter** (char) - The opening delimiter character of the block.
- **block.ClosingDelimiter** (char) - The closing delimiter character of the block.
- **block.Type** (TokenType) - An enum indicating the type of block (e.g., BraceBlock, BracketBlock, ParenthesisBlock).
- **block.Children** (Array<IToken>) - An array of tokens representing the parsed content within the block.

#### Response Example
```json
{
  "fullContent": "{inner content}",
  "innerContent": "inner content",
  "openingDelimiter": "{",
  "closingDelimiter": "}",
  "type": "BraceBlock",
  "children": [
    {
      "type": "TextToken",
      "content": "inner"
    },
    {
      "type": "WhitespaceToken",
      "content": " "
    },
    {
      "type": "TextToken",
      "content": "content"
    }
  ]
}
```
```

--------------------------------

### Create Tokenizer and Parse Text

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Demonstrates basic tokenization of a string into an immutable array of tokens using default options. The tokenizer processes input using Memory<char> and returns an array of token objects.

```csharp
using TinyTokenizer;

// Parse a simple function call
var source = "func(a, b)";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

// Result: 2 tokens
// tokens[0]: TextToken("func")
// tokens[1]: BlockToken("(a, b)") with 4 children:
//   - TextToken("a")
//   - SymbolToken(",")
//   - WhitespaceToken(" ")
//   - TextToken("b")

Console.WriteLine($"Found {tokens.Length} top-level tokens");
```

--------------------------------

### Configure Symbol Recognition with TokenizerOptions in C#

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Demonstrates how to customize symbol tokens in TinyTokenizer. It shows adding, removing, and replacing the default set of symbols, affecting how input strings are tokenized. This allows fine-grained control over what characters are treated as delimiters.

```csharp
using TinyTokenizer;

// Default symbols: / : , ; = + - * < > ! & | . @ # ? % ^ ~ \
var defaultOptions = TokenizerOptions.Default;

// Add additional symbols
var withCustom = defaultOptions.WithAdditionalSymbols('$', '_');
var tokenizer1 = new Tokenizer("$price_value".AsMemory(), withCustom);
var tokens1 = tokenizer1.Tokenize();
// Result: SymbolToken($), TextToken(price), SymbolToken(_), TextToken(value)

// Remove symbols so they become part of text
var withoutSlash = defaultOptions.WithoutSymbols('/');
var tokenizer2 = new Tokenizer("path/to/file".AsMemory(), withoutSlash);
var tokens2 = tokenizer2.Tokenize();
// Result: TextToken("path/to/file") - single token without / being a separator

// Replace entire symbol set
var minimal = defaultOptions.WithSymbols(':', ',', ';');
var tokenizer3 = new Tokenizer("a:b,c;d+e".AsMemory(), minimal);
var tokens3 = tokenizer3.Tokenize();
// Result: TextToken(a), SymbolToken(:), TextToken(b), SymbolToken(,
//         TextToken(c), SymbolToken(;), TextToken(d+e)

foreach (var token in tokens3)
{
    if (token is SymbolToken symbol)
        Console.WriteLine($"Symbol: {symbol.Symbol}");
    else if (token is TextToken text)
        Console.WriteLine($"Text: {text.ContentSpan.ToString()}");
}
```

--------------------------------

### Working with Token Types and Properties in C#

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

This C# snippet demonstrates how to tokenize a source string and then pattern match on the different token types (TextToken, BlockToken, SymbolToken, WhitespaceToken, ErrorToken) to extract their properties and content. It utilizes TinyTokenizer's capabilities to break down a simple expression.

```csharp
using TinyTokenizer;

var source = "func(x, y) + data[0]";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

foreach (var token in tokens)
{
    switch (token)
    {
        case TextToken text:
            Console.WriteLine($"Text: '{text.ContentSpan.ToString()}'");
            break;

        case BlockToken block:
            Console.WriteLine($"Block: {block.Type} with {block.Children.Length} children");
            Console.WriteLine($"  Opens with: {block.OpeningDelimiter}");
            Console.WriteLine($"  Closes with: {block.ClosingDelimiter}");
            break;

        case SymbolToken symbol:
            Console.WriteLine($"Symbol: {symbol.Symbol}");
            break;

        case WhitespaceToken ws:
            Console.WriteLine($"Whitespace: {ws.ContentSpan.Length} characters");
            break;

        case ErrorToken error:
            Console.WriteLine($"Error: {error.ErrorMessage}");
            break;
    }
}

// Output:
// Text: 'func'
// Block: ParenthesisBlock with 4 children
//   Opens with: (
//   Closes with: )
// Whitespace: 1 characters
// Symbol: +
// Whitespace: 1 characters
// Text: 'data'
// Block: BracketBlock with 1 children
//   Opens with: [
//   Closes with: ]
```

--------------------------------

### Handle Errors with TinyTokenizer in C#

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Illustrates error handling and recovery mechanisms in TinyTokenizer. It shows how the tokenizer can continue parsing even when encountering malformed input, and how to inspect the resulting tokens for errors. This is crucial for robust parsing of potentially invalid data.

```csharp
using TinyTokenizer;

var source = "}hello{world";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

// Tokenizer continues after errors
// tokens[0]: ErrorToken("}", "Unexpected closing delimiter '}'", position: 0)
// tokens[1]: TextToken("hello")
// tokens[2]: ErrorToken("{", "Unclosed block starting with '{'", position: 6)
// tokens[3]: TextToken("world")

// Check if any errors exist
if (tokens.HasErrors())
{
    Console.WriteLine("Parsing errors detected!");

    // Retrieve all errors (including nested ones)
    foreach (var error in tokens.GetErrors())
    {
        Console.WriteLine($"Error at position {error.Position}: {error.ErrorMessage}");
        Console.WriteLine($"  Content: '{error.ContentSpan.ToString()}'");
    }
}

// Output:
// Parsing errors detected!
// Error at position 0: Unexpected closing delimiter '}'
//   Content: '}'
// Error at position 6: Unclosed block starting with '{'
//   Content: '{'
```

--------------------------------

### Parsing Nested Blocks

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Illustrates the recursive parsing of deeply nested declaration blocks with multiple levels.

```APIDOC
## Parsing Nested Blocks

### Description
Recursive parsing of deeply nested declaration blocks with multiple levels.

### Method
POST (Conceptual - this is a library usage example, not a REST endpoint)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```csharp
using TinyTokenizer;

var source = "{outer [inner (deepest)]}";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

// Navigate the nested structure
var braceBlock = (BlockToken)tokens[0];
Console.WriteLine($"Level 1: {braceBlock.FullContent.ToString()}");

// braceBlock.Children contains: TextToken("outer"), WhitespaceToken(" "), BlockToken("[inner (deepest)]")
var bracketBlock = (BlockToken)braceBlock.Children[2];
Console.WriteLine($"Level 2: {bracketBlock.FullContent.ToString()}");

// bracketBlock.Children contains: TextToken("inner"), WhitespaceToken(" "), BlockToken("(deepest)")
var parenBlock = (BlockToken)bracketBlock.Children[2];
Console.WriteLine($"Level 3: {parenBlock.FullContent.ToString()}");
Console.WriteLine($"Deepest content: {parenBlock.InnerContent.ToString()}");

// Output:
// Level 1: {outer [inner (deepest)]}
// Level 2: [inner (deepest)]
// Level 3: (deepest)
// Deepest content: deepest
```

### Response
#### Success Response (200)
- **tokens** (Array<IToken>) - An array of top-level tokens. Nested `BlockToken` objects will contain their children in their respective `Children` property.

#### Response Example
```json
{
  "tokens": [
    {
      "type": "BlockToken",
      "content": "{outer [inner (deepest)]}",
      "openingDelimiter": "{",
      "closingDelimiter": "}",
      "children": [
        {
          "type": "TextToken",
          "content": "outer"
        },
        {
          "type": "WhitespaceToken",
          "content": " "
        },
        {
          "type": "BlockToken",
          "content": "[inner (deepest)]",
          "openingDelimiter": "[",
          "closingDelimiter": "]",
          "children": [
            {
              "type": "TextToken",
              "content": "inner"
            },
            {
              "type": "WhitespaceToken",
              "content": " "
            },
            {
              "type": "BlockToken",
              "content": "(deepest)",
              "openingDelimiter": "(",
              "closingDelimiter": ")",
              "children": [
                {
                  "type": "TextToken",
                  "content": "deepest"
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}
```
```

--------------------------------

### Filter Tokens by Type using Extensions in C#

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Shows how to use extension methods provided by TinyTokenizer to filter and retrieve specific types of tokens from a parsed sequence. This includes finding text, block, and symbol tokens, simplifying post-parsing analysis and manipulation.

```csharp
using TinyTokenizer;

var source = "{func(a, b)} [array] text:value";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

// Get all text tokens including nested ones
var textTokens = tokens.OfTokenType<TextToken>().ToList();
foreach (var text in textTokens)
{
    Console.WriteLine($"Text: {text.ContentSpan.ToString()}");
}
// Output: Text: func
//         Text: a
//         Text: b
//         Text: array
//         Text: text
//         Text: value

// Get all block tokens
var blocks = tokens.OfTokenType<BlockToken>().ToList();
Console.WriteLine($"Found {blocks.Count} blocks");
foreach (var block in blocks)
{
    Console.WriteLine($"  {block.Type}: {block.FullContent.ToString()}");
}
// Output: Found 2 blocks
//           ParenthesisBlock: (a, b)
//           BracketBlock: [array]

// Get all symbols
var symbols = tokens.OfTokenType<SymbolToken>().ToList();
foreach (var symbol in symbols)
{
    Console.WriteLine($"Symbol: {symbol.Symbol}");
}
// Output: Symbol: :

```

--------------------------------

### Access BlockToken Properties

Source: https://context7.com/dsisco11/tinytokenizer/llms.txt

Illustrates how to access the content, delimiters, and child tokens of a BlockToken. This allows inspection of the structure within parsed blocks, including nested elements.

```csharp
using TinyTokenizer;

var source = "{inner content}";
var tokenizer = new Tokenizer(source.AsMemory());
var tokens = tokenizer.Tokenize();

var block = (BlockToken)tokens[0];

// Full content includes delimiters
Console.WriteLine(block.FullContent.ToString());      // "{inner content}"

// Inner content excludes delimiters
Console.WriteLine(block.InnerContent.ToString());     // "inner content"

// Access delimiter characters
Console.WriteLine(block.OpeningDelimiter);            // '{'
Console.WriteLine(block.ClosingDelimiter);            // '}'

// Token type identifies the block type
Console.WriteLine(block.Type);                        // TokenType.BraceBlock

// Children are recursively parsed tokens
foreach (var child in block.Children)
{
    Console.WriteLine($"{child.GetType().Name}: {child.ContentSpan.ToString()}");
}
// Output: TextToken: inner
//         WhitespaceToken: 
//         TextToken: content
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.