### Parse Token List Example Source: https://github.com/datalust/superpower/blob/dev/_autodocs/02-result-types.md An example demonstrating parsing a token stream for a specific token kind, such as ArithmeticToken.Plus. It checks the result and reports success or failure details. ```csharp enum ArithmeticToken { Plus, Minus, Number } var parser = Token.EqualTo(ArithmeticToken.Plus); var result = parser(tokenList); if (result.HasValue) { Console.WriteLine($"Matched: {result.Value.Kind}"); } else { Console.WriteLine($"Error: {result.ErrorMessage}"); Console.WriteLine($"At token position: {result.ErrorTokenPosition}"); } ``` -------------------------------- ### Example: Parsing Words and Whitespace Source: https://github.com/datalust/superpower/blob/dev/_autodocs/03-combinators.md Demonstrates using `AtLeastOnce` to parse a word and `Many` to parse whitespace characters. ```csharp // Parse one or more letters var word = Character.Letter.AtLeastOnce(); word.Parse("hello"); // Returns ['h', 'e', 'l', 'l', 'o'] // Parse zero or more whitespace characters var spaces = Character.WhiteSpace.Many(); spaces.Parse(" x"); // Returns [' ', ' ', ' '] ``` -------------------------------- ### Add Superpower Package Source: https://github.com/datalust/superpower/blob/dev/README.md Install the Superpower NuGet package using the .NET CLI. ```shell dotnet add package Superpower ``` -------------------------------- ### Example: Repeating Digits and Hex Digits Source: https://github.com/datalust/superpower/blob/dev/_autodocs/03-combinators.md Shows how to use `Repeat` for digits and `RepeatString` for hexadecimal characters. ```csharp // Parse exactly 3 digits var threeDig = Character.Digit.Repeat(3); threeDig.Parse("123abc"); // Returns ['1', '2', '3'] // Parse exactly 4 hex digits var hex = Character.HexDigit.RepeatString(4); hex.Parse("ABCD xyz"); // Returns "ABCD" ``` -------------------------------- ### Attribute for Token Category and Example Source: https://github.com/datalust/superpower/blob/dev/README.md Illustrates how to use the `[Token]` attribute to provide metadata for a token type, such as its category and an example, which can improve error reporting. ```csharp public enum ArithmeticExpressionToken { None, Number, [Token(Category = "operator", Example = "+")] Plus, ``` -------------------------------- ### Tokenization and Parsing Example Source: https://github.com/datalust/superpower/blob/dev/README.md Demonstrates the two-step process of tokenization and parsing for an arithmetic expression. First, tokenize the input string, then parse the resulting token list. ```csharp var expression = "1 * (2 + 3)"; // 1. var tokenizer = new ArithmeticExpressionTokenizer(); var tokenList = tokenizer.Tokenize(expression); // 2. var parser = ArithmeticExpressionParser.Lambda; // parser built with combinators var expressionTree = parser.Parse(tokenList); // Use the result var eval = expressionTree.Compile(); Console.WriteLine(eval()); // -> 5 ``` -------------------------------- ### Minimal Tokenization and Parsing Setup Source: https://github.com/datalust/superpower/blob/dev/_autodocs/00-index.md Shows the basic steps for tokenizing input text and preparing for parsing. This involves building a tokenizer with specific rules and then tokenizing the input string. ```csharp // Step 1: Tokenize var tokenizer = new TokenizerBuilder() .Ignore(Span.WhiteSpace) .Match(Numerics.Natural, TokenKind.Number) .Match(Character.EqualTo('+'), TokenKind.Plus) .Build(); var tokens = tokenizer.Tokenize("1 + 2"); // Step 2: Parse var parser = /* define grammar */; var result = parser.Parse(tokens); ``` -------------------------------- ### Example: Matching Operator and Number Tokens Source: https://github.com/datalust/superpower/blob/dev/_autodocs/08-token-parsers.md Demonstrates how to use the Matching method to create parsers for different token kinds like operators and numbers using predicate functions. ```csharp public enum TokenKind { Number, Plus, Minus, Multiply, Divide } // Match any operator token var operator = Token.Matching(k => k is TokenKind.Plus or TokenKind.Minus or TokenKind.Multiply or TokenKind.Divide, "operator"); // Match number tokens var number = Token.Matching(k => k == TokenKind.Number, "number"); // Match tokens by numeric kind (if using numeric enums) var arithmeticOp = Token.Matching(k => ((int)(object)k) >= 1 && ((int)(object)k) <= 4, "arithmetic operator"); ``` -------------------------------- ### Example: Optional Sign and Decimal Part Source: https://github.com/datalust/superpower/blob/dev/_autodocs/03-combinators.md Illustrates using `IfMissing` for an optional sign and `Optional` for an optional decimal part. ```csharp // Parse optional sign var sign = Character.EqualTo('-').IfMissing('+'); sign.Parse("-5"); // Returns '-' sign.Parse("5"); // Returns '+' (fallback) // Parse optional decimal part var optionalDecimal = Character.EqualTo('.').Optional(); ``` -------------------------------- ### Sequence Parser Example (C#) Source: https://github.com/datalust/superpower/blob/dev/_autodocs/04-parse-helpers.md Demonstrates parsing three distinct elements (letter, digit, letter) and retrieving their values using the Sequence combinator. ```csharp // Parse three separate elements and get all three var (first, second, third) = Parse.Sequence( Character.Letter, Character.Digit, Character.Letter ).Parse("a1b"); // Equivalent to: from c1 in Character.Letter from c2 in Character.Digit from c3 in Character.Letter select (c1, c2, c3) ``` -------------------------------- ### Token Parsers Source: https://github.com/datalust/superpower/blob/dev/_autodocs/13-quick-reference.md Examples of matching tokens by kind, value, or a predicate. Used for precise token stream analysis. ```csharp Token.EqualTo(TokenKind.Plus) // Exact kind match Token.EqualToValue(Kind.Id, "if") // Kind and value Token.EqualToValueIgnoreCase(Kind.Id, "if") Token.Matching(k => true, "name") // Predicate Token.Sequence(K1, K2, K3) // Exact sequence ``` -------------------------------- ### Until Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Creates a new TextSpan from the start of the current span up to the position of another specified span. ```APIDOC ## Until ### Description Span from start to another span's position. ### Method Signature ```csharp public TextSpan Until(TextSpan next) ``` ### Parameters #### Path Parameters - **next** (TextSpan) - Required - Another span in same source ### Returns `TextSpan` — Span from start to `next`'s position ### Example ```csharp var source = new TextSpan("hello world"); var beforeSpace = source.Until(source.Skip(5)); // beforeSpace covers "hello" ``` ``` -------------------------------- ### Arithmetic Expression Tokenizer Example Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Builds a tokenizer for arithmetic expressions using TokenizerBuilder. This tokenizer can recognize numbers, operators, and parentheses. ```csharp public enum ArithmeticToken { Number, Plus, Minus, Multiply, Divide, LeftParen, RightParen } var tokenizer = new TokenizerBuilder() .Ignore(Span.WhiteSpace) .Match(Character.EqualTo('+'), ArithmeticToken.Plus, "+") .Match(Character.EqualTo('-'), ArithmeticToken.Minus, "-") .Match(Character.EqualTo('*'), ArithmeticToken.Multiply, "*") .Match(Character.EqualTo('/'), ArithmeticToken.Divide, "/") .Match(Character.EqualTo('('), ArithmeticToken.LeftParen, "(") .Match(Character.EqualTo(')'), ArithmeticToken.RightParen, ")") .Match(Numerics.Natural, ArithmeticToken.Number) .Build(); var result = tokenizer.TryTokenize("1 + 2 * 3"); if (result.HasValue) { var tokens = result.Value; // Use for parsing } ``` -------------------------------- ### Custom Tokenizer Implementation Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Example of creating a custom tokenizer by inheriting from the Tokenizer base class and implementing the Tokenize method. This is used for defining how input text is converted into a list of tokens. ```csharp public class MyTokenizer : Tokenizer { protected override IEnumerable> Tokenize(TextSpan span) { // Implementation } } var tokenizer = new MyTokenizer(); var tokens = tokenizer.Tokenize("input text"); ``` -------------------------------- ### Parse Arithmetic Expression with Error Reporting Source: https://github.com/datalust/superpower/blob/dev/README.md Demonstrates parsing an arithmetic expression using the defined parser and shows an example of the resulting syntax error message. This highlights Superpower's error reporting capabilities. ```csharp ArithmeticExpressionParser.Lambda.Parse(new ArithmeticExpressionTokenizer().Tokenize("1 + * 3")); // -> Syntax error (line 1, column 5): unexpected operator `*`, expected expression. ``` -------------------------------- ### Recursive Parser Example (C#) Source: https://github.com/datalust/superpower/blob/dev/_autodocs/04-parse-helpers.md Defines a recursive grammar for expressions that can include parenthesized sub-expressions. The parser is evaluated once and cached. ```csharp // Define an expression that can contain parenthesized expressions TokenListParser Expr = null!; var Factor = (from lparen in Token.EqualTo(Token.LParen) from expr in Parse.Ref(() => Expr) from rparen in Token.EqualTo(Token.RParen) select expr) .Or(Number); Expr = Parse.Chain(Multiply.Or(Divide), Factor, MakeBinary); // Expr can now reference itself through Factor ``` -------------------------------- ### Parse Identifiers Source: https://github.com/datalust/superpower/blob/dev/README.md Construct a parser for identifiers using a combination of character parsers and combinators. This example parses a letter followed by zero or more letters, digits, or underscores. ```csharp TextParser identifier = from first in Character.Letter from rest in Character.LetterOrDigit.Or(Character.EqualTo('_')).Many() select first + new string(rest); var id = identifier.Parse("abc123"); Assert.Equal("abc123", id); ``` -------------------------------- ### C-style Identifier Parser Source: https://github.com/datalust/superpower/blob/dev/_autodocs/05-character-parsers.md Parses C-style identifiers which start with a letter or underscore, followed by letters, digits, or underscores. Requires no special setup. ```csharp // C-style identifier: letter or underscore, followed by letters, digits, or underscores var identifier = Character.Letter.Or(Character.EqualTo('_')).Then(first => Character.LetterOrDigit.Or(Character.EqualTo('_')).Many() .Select(rest => first + new string(rest)) ); ``` -------------------------------- ### Getting Error Position Source: https://github.com/datalust/superpower/blob/dev/_autodocs/13-quick-reference.md Access the `ErrorPosition` property on a failed parse result to get the line and column number of the error. ```csharp var pos = result.ErrorPosition; Console.WriteLine($"Line {pos.Line}, Column {pos.Column}"); ``` -------------------------------- ### Parse Character Digit Example Source: https://github.com/datalust/superpower/blob/dev/_autodocs/02-result-types.md An example of parsing a single digit from a text span using Character.Digit. It checks the result for success or failure and prints relevant information. ```csharp var parser = Character.Digit; var result = parser(new TextSpan("5abc")); if (result.HasValue) { Console.WriteLine($"Parsed: {result.Value}"); Console.WriteLine($"Next position: {result.Remainder}"); } else { Console.WriteLine(result.ErrorMessage); Console.WriteLine($"Error at: {result.ErrorPosition}"); } ``` -------------------------------- ### Token Parsing with Full Flow Source: https://github.com/datalust/superpower/blob/dev/_autodocs/00-index.md Illustrates the complete process of tokenizing source code and then parsing the tokens into an Abstract Syntax Tree (AST). ```csharp // Token parsing with full flow var tokens = tokenizer.Tokenize(sourceCode); var ast = expressionParser.Parse(tokens); ``` -------------------------------- ### Parse a Word Source: https://github.com/datalust/superpower/blob/dev/_autodocs/13-quick-reference.md Define a parser for words starting with a letter, followed by zero or more letters or digits. ```csharp var word = Character.Letter .Then(c1 => Character.LetterOrDigit.Many() .Select(rest => c1 + new string(rest)) ); ``` -------------------------------- ### Create a Position Instance Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Demonstrates how to create new Position instances using the constructor. Positions track absolute character index, line number, and column number. ```csharp var start = new Position(0, 1, 1); var middle = new Position(10, 1, 11); var later = new Position(50, 3, 15); ``` -------------------------------- ### Create TextSpan Instances Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Demonstrates the creation of TextSpan objects using different constructors. The first creates a span for the entire string, while the second creates a span for a specific substring. ```csharp var fullSpan = new TextSpan("hello world"); var substring = new TextSpan("hello world", new Position(6, 1, 7), 5); // substring covers "world" ``` -------------------------------- ### Identifier Parsers Source: https://github.com/datalust/superpower/blob/dev/_autodocs/14-all-types.md Defines a parser for C-style identifiers, which start with a letter or underscore and are followed by letters, digits, or underscores. ```APIDOC ## Identifier Parsers ### Description Defines a parser for C-style identifiers, which start with a letter or underscore and are followed by letters, digits, or underscores. ### Parsers - `CStyle` → TextSpan — [a-zA-Z_][a-zA-Z0-9_]* ``` -------------------------------- ### Create a TokenList Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Initialize a TokenList with an array of tokens. This structure manages the sequence of tokens during parsing. ```csharp var tokens = new[] { new Token(TokenKind.Plus, ...), new Token(TokenKind.Number, ...) }; var tokenList = new TokenList(tokens); ``` -------------------------------- ### TextSpan Extensions and LINQ Combination Source: https://github.com/datalust/superpower/blob/dev/_autodocs/07-span-and-text-parsers.md Demonstrates how to convert TextSpans to strings and combine parsers using LINQ for more complex parsing scenarios. ```APIDOC ## TextSpan Extensions ### Description All span parsers return `TextSpan`. This section shows how to convert them to strings and combine parsers using LINQ. ### Convert TextSpan to String ```csharp TextSpan span = ...; string value = span.ToStringValue(); ``` ### Combine Parsers with LINQ ```csharp var result = from keyword in Span.EqualTo("while") from ws in Span.Whitespace from condition in Span.AnyCharExcept(')') select (keyword.ToStringValue(), condition.ToStringValue()); ``` ### Related Parsers See [[05-character-parsers.md]] and [[06-numerics-parsers.md]] for character and numeric parsing. ``` -------------------------------- ### Safe Tokenization with TryTokenize Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Demonstrates using the TryTokenize method for safe tokenization, which returns a Result object instead of throwing an exception on failure. This is useful when explicit error handling is required. ```csharp var result = tokenizer.TryTokenize("input"); if (result.HasValue) { var tokens = result.Value; // Process tokens } else { Console.WriteLine(result.ErrorMessage); } ``` -------------------------------- ### SqlLineComment Parser Source: https://github.com/datalust/superpower/blob/dev/_autodocs/07-span-and-text-parsers.md Parses SQL single-line comments, which start with -- and continue to the end of the line. It returns the TextSpan of the comment. ```APIDOC ## SqlLineComment Parse SQL single-line comment. ```csharp public static TextParser SqlLineComment { get; } ``` **Returns:** `TextParser` — Span of SQL comment **Format:** `-- text` to end of line **Example:** ```csharp var sql = Comment.SqlLineComment; sql.Parse("-- select * from users\nREST"); // TextSpan for "-- select * from users" ``` ``` -------------------------------- ### CLineComment Parser Source: https://github.com/datalust/superpower/blob/dev/_autodocs/07-span-and-text-parsers.md Parses only C-style single-line comments, starting with // and extending to the end of the line. It returns the TextSpan of the comment. ```APIDOC ## CLineComment Parse C-style single-line comment only. ```csharp public static TextParser CLineComment { get; } ``` **Returns:** `TextParser` — Span of single-line comment **Format:** `// text` to end of line **Example:** ```csharp var line = Comment.CLineComment; line.Parse("// this is a comment\nrest"); // TextSpan for "// this is a comment" ``` ``` -------------------------------- ### Create a Token Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Instantiate a Token with its kind and text span. Ensure the TokenKind enum and TextSpan are defined. ```csharp public enum TokenKind { Plus, Number } var plusToken = new Token( TokenKind.Plus, new TextSpan("+") ); ``` -------------------------------- ### Get Token Value as String Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Retrieve the source text associated with a token. This method is useful for displaying or logging token content. ```csharp var text = token.ToStringValue(); ``` -------------------------------- ### Minimal Integer Parsing Source: https://github.com/datalust/superpower/blob/dev/_autodocs/00-index.md Demonstrates how to parse an integer directly using a pre-built numeric parser. This is useful for simple numeric conversions. ```csharp // Parse a number directly TextParser parser = Numerics.IntegerInt32; int value = parser.Parse("42"); // Returns 42 ``` -------------------------------- ### Recognizer Order in Tokenization Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Demonstrates the correct and incorrect ordering of recognizers. Place more specific patterns before general ones to ensure they are matched correctly. ```csharp // Good: specific keywords before general identifiers .Match(Span.EqualTo("if"), TokenKind.IfKeyword) .Match(Identifier.CStyle, TokenKind.Identifier) // Bad: identifier would match "if" first .Match(Identifier.CStyle, TokenKind.Identifier) .Match(Span.EqualTo("if"), TokenKind.IfKeyword) ``` -------------------------------- ### Parse SQL Single-Line Comments Source: https://github.com/datalust/superpower/blob/dev/_autodocs/07-span-and-text-parsers.md Use `Comment.SqlLineComment` to parse SQL single-line comments that start with '--'. It returns the span of the comment. ```csharp var sql = Comment.SqlLineComment; sql.Parse("-- select * from users\nREST"); // TextSpan for "-- select * from users" ``` -------------------------------- ### Consume Next Token from TokenList Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Extract the next token from the TokenList and get the remaining tokens. Check 'HasValue' to determine if a token was available. ```csharp var list = new TokenList(tokens); var result = list.ConsumeToken(); if (result.HasValue) { var token = result.Value; var rest = result.Remainder; } ``` -------------------------------- ### Consume Character from TextSpan Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Shows how to extract the first character from a TextSpan and get the remaining span. The result contains the character and the updated span. ```csharp var span = new TextSpan("abc"); var result = span.ConsumeChar(); // result.Value == 'a' // result.Remainder spans "bc" ``` -------------------------------- ### TextSpan Static Instances Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Provides static instances for common TextSpan states. ```APIDOC ## TextSpan Static Instances ### Description Provides static instances for common TextSpan states. | Member | Description | |--------|-------------| | TextSpan.None | Uninitialized span (Source = null) | | TextSpan.Empty | Empty initialized span (Source = "", Length = 0) | ``` -------------------------------- ### Parse C-Style Single-Line Comments Source: https://github.com/datalust/superpower/blob/dev/_autodocs/07-span-and-text-parsers.md Use `Comment.CLineComment` to specifically parse C-style single-line comments starting with '//'. It returns the span of the comment. ```csharp var line = Comment.CLineComment; line.Parse("// this is a comment\nrest"); // TextSpan for "// this is a comment" ``` -------------------------------- ### Two-Phase Parsing: Tokenize then Parse Source: https://github.com/datalust/superpower/blob/dev/_autodocs/11-parser-extensions.md Implement a two-phase parsing strategy where the input is first tokenized, and then the resulting tokens are parsed into an Abstract Syntax Tree (AST). Handle errors at each phase. ```csharp // Phase 1: Tokenize var tokenizer = new MyTokenizer(); var tokenResult = tokenizer.TryTokenize(sourceCode); if (!tokenResult.HasValue) { Console.WriteLine($"Tokenization error: {tokenResult}"); return; } // Phase 2: Parse var parser = MyExpressionParser.Expression; var parseResult = parser.TryParse(tokenResult.Value); if (!parseResult.HasValue) { Console.WriteLine($"Parse error: {parseResult}"); return; } var ast = parseResult.Value; // Process AST ``` -------------------------------- ### Parse Currency Amount Source: https://github.com/datalust/superpower/blob/dev/_autodocs/06-numerics-parsers.md Parse a currency amount starting with a dollar sign, followed by a decimal number. The output is rounded to two decimal places. ```csharp var currency = Character.EqualTo('$').IgnoreThen( Numerics.Decimal .Select(d => Math.Round(d, 2)) ); currency.Parse("$19.99 USD"); // 19.99m ``` -------------------------------- ### Instant Parsers Source: https://github.com/datalust/superpower/blob/dev/_autodocs/14-all-types.md Provides a parser for ISO 8601 formatted date-time strings, returning a DateTimeOffset object. ```APIDOC ## Instant Parsers ### Description Provides a parser for ISO 8601 formatted date-time strings, returning a DateTimeOffset object. ### Parsers - `ISO8601DateTime` → DateTimeOffset — ISO 8601 format ``` -------------------------------- ### Create Success and Failure Results Source: https://github.com/datalust/superpower/blob/dev/_autodocs/02-result-types.md Demonstrates how to manually construct success and failure results for character-based parsing. Use Result.Value for successful parses and Result.Empty for failures, optionally providing expectations. ```csharp Result.Value(parsedValue, inputLocation, remainder) Result.Empty(input, expectations: new[] { "expected text" }) ``` -------------------------------- ### Build Tokenizer with Ignore and Match Rules Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Construct a tokenizer using TokenizerBuilder. Use Ignore to skip specific text patterns like whitespace or comments, and Match to produce tokens for recognized patterns. ```csharp var tokenizer = new TokenizerBuilder() .Ignore(Span.WhiteSpace) .Match(Character.EqualTo('+'), TokenKind.Plus) .Match(Character.EqualTo('-'), TokenKind.Minus) .Match(Numerics.Natural, TokenKind.Number) .Build(); ``` ```csharp .Ignore(Span.WhiteSpace) .Ignore(Comment.CStyle) ``` ```csharp .Match(Numerics.Natural, TokenKind.Number) .Match(Identifier.CStyle, TokenKind.Identifier) .Match(Span.EqualTo("=>"), TokenKind.Arrow) ``` ```csharp var tokenizer = new TokenizerBuilder() .Ignore(Span.WhiteSpace) .Match(Character.EqualTo('+'), ArithmeticToken.Plus) .Match(Character.EqualTo('-'), ArithmeticToken.Minus) .Match(Character.EqualTo('*'), ArithmeticToken.Multiply) .Match(Character.EqualTo('/'), ArithmeticToken.Divide) .Match(Numerics.Natural, ArithmeticToken.Number) .Build(); ``` -------------------------------- ### Accessing Error Position Information from ParseException Source: https://github.com/datalust/superpower/blob/dev/_autodocs/12-exceptions.md When a ParseException is caught, you can access the ErrorPosition property to get line, column, and absolute character offset information about the parsing error. ```csharp catch (ParseException ex) { Console.WriteLine($"Line: {ex.ErrorPosition.Line}"); Console.WriteLine($"Column: {ex.ErrorPosition.Column}"); Console.WriteLine($"Absolute: {ex.ErrorPosition.Absolute}"); } ``` -------------------------------- ### TokenizerBuilder Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md A fluent builder for assembling tokenizers from recognizer rules. ```APIDOC ## TokenizerBuilder **Namespace:** `Superpower` **Class:** `public class TokenizerBuilder` Fluent builder for assembling tokenizers from recognizer rules. ### Building a Tokenizer ```csharp var tokenizer = new TokenizerBuilder() .Ignore(Span.WhiteSpace) .Match(Character.EqualTo('+'), TokenKind.Plus) .Match(Character.EqualTo('-'), TokenKind.Minus) .Match(Numerics.Natural, TokenKind.Number) .Build(); ``` ### Builder Methods #### Ignore Skip text matching a parser without producing tokens. ```csharp public TokenizerBuilder Ignore(TextParser recognizer) ``` ##### Parameters - **recognizer** (TextParser) - Required - Parser for text to skip ##### Returns `TokenizerBuilder` — This builder for chaining ##### Example ```csharp .Ignore(Span.WhiteSpace) .Ignore(Comment.CStyle) ``` #### Match Produce a token for matched text. ```csharp public TokenizerBuilder Match( TextParser recognizer, TKind kind) public TokenizerBuilder Match( TextParser recognizer, TKind kind, string example) ``` ##### Parameters - **recognizer** (TextParser) - Required - Parser for text to recognize - **kind** (TKind) - Required - Token kind to produce - **example** (string) - Optional - Example for error messages ##### Returns `TokenizerBuilder` — This builder for chaining ##### Example ```csharp .Match(Numerics.Natural, TokenKind.Number) .Match(Identifier.CStyle, TokenKind.Identifier) .Match(Span.EqualTo("=>"), TokenKind.Arrow) ``` #### Build Create the tokenizer. ```csharp public Tokenizer Build() ``` ##### Returns `Tokenizer` — Configured tokenizer instance ##### Example ```csharp var tokenizer = new TokenizerBuilder() .Ignore(Span.WhiteSpace) .Match(Character.EqualTo('+'), ArithmeticToken.Plus) .Match(Character.EqualTo('-'), ArithmeticToken.Minus) .Match(Character.EqualTo('*'), ArithmeticToken.Multiply) .Match(Character.EqualTo('/'), ArithmeticToken.Divide) .Match(Numerics.Natural, ArithmeticToken.Number) .Build(); ``` ``` -------------------------------- ### Build a Quick Tokenizer Source: https://github.com/datalust/superpower/blob/dev/_autodocs/13-quick-reference.md Use TokenizerBuilder to define rules for ignoring whitespace and comments, and matching specific tokens like arrows, identifiers, and numbers. ```csharp var tokenizer = new TokenizerBuilder() .Ignore(Span.WhiteSpace) .Ignore(Comment.CStyle) .Match(Span.EqualTo("=>"), TokenKind.Arrow) .Match(Identifier.CStyle, TokenKind.Identifier) .Match(Numerics.Natural, TokenKind.Number) .Build(); ``` -------------------------------- ### In Source: https://github.com/datalust/superpower/blob/dev/_autodocs/05-character-parsers.md Creates a parser that matches any character from a given set. ```APIDOC ## In ### Description Match any character from a set. ### Method Signature ```csharp public static TextParser In(params char[] chars) ``` ### Parameters #### Path Parameters - **chars** (char[]) - Required - Characters to match ### Returns `TextParser` — Parser matching any character in the set ### Example ```csharp var operator = Character.In('+', '-', '*', '/'); operator.Parse("*5"); // Returns '*' var vowel = Character.In('a', 'e', 'i', 'o', 'u'); vowel.Parse("apple"); // Returns 'a' ``` ``` -------------------------------- ### TextSpan Struct Source: https://github.com/datalust/superpower/blob/dev/_autodocs/14-all-types.md Represents a contiguous span of text within a source, defined by a starting position and a length. It supports operations like consuming characters, extracting sub-spans, and converting to a string. ```APIDOC ## TextSpan Struct ### Description Represents a contiguous span of text within a source, defined by a starting position and a length. It supports operations like consuming characters, extracting sub-spans, and converting to a string. ### Properties - `Source` (string?) - The source text from which this span originates. - `Position` (Position) - The starting position of the span. - `Length` (int) - The length of the span in characters. - `IsAtEnd` (bool) - Indicates if the span is at the end of its source. ### Constructors - `TextSpan(string source)` - Creates a TextSpan covering the entire source string. - `TextSpan(string source, Position position, int length)` - Creates a TextSpan with a specific source, position, and length. ### Static Members - `None` - Represents an uninitialized TextSpan. - `Empty` - Represents an initialized, empty TextSpan. ### Methods - `ConsumeChar()` - Consumes and returns the first character of the span, advancing the span. - `Until(TextSpan)` - Returns a new TextSpan representing the text up to the start of another TextSpan. - `First(int)` - Returns a new TextSpan containing the first specified number of characters. - `Skip(int)` - Returns a new TextSpan with the specified number of characters skipped from the beginning. - `ToStringValue()` - Returns the string value of the TextSpan. ``` -------------------------------- ### Handling Invalid Zero-Width Tokens in Custom Tokenizer Source: https://github.com/datalust/superpower/blob/dev/_autodocs/12-exceptions.md A custom tokenizer should ensure each yielded token consumes at least one character to prevent ParseException. This example shows a problematic implementation that yields zero-width tokens. ```csharp protected override IEnumerable> Tokenize(TextSpan span) { // BUG: This advances span but yields the same position yield return Result.Value(TokenKind.Keyword, span); // Throws ParseException: "Zero-width tokens are not supported" } ``` -------------------------------- ### TokenizerBuilder Class Methods Source: https://github.com/datalust/superpower/blob/dev/_autodocs/14-all-types.md A builder class for constructing Tokenizer instances. It allows defining rules for ignoring text, matching specific patterns to token kinds, and finally building the tokenizer. ```csharp public class TokenizerBuilder ``` -------------------------------- ### Build Parser for Binary Operators Source: https://github.com/datalust/superpower/blob/dev/_autodocs/08-token-parsers.md Combines individual token parsers for different binary operators into a single parser. Use this to create a parser that can recognize any of the specified arithmetic operators. ```csharp public enum TokenKind { Plus, Minus, Star, Slash } var plusParser = Token.EqualTo(TokenKind.Plus).Value(ExpressionType.Add); var minusParser = Token.EqualTo(TokenKind.Minus).Value(ExpressionType.Subtract); var multiplyParser = Token.EqualTo(TokenKind.Star).Value(ExpressionType.Multiply); var divideParser = Token.EqualTo(TokenKind.Slash).Value(ExpressionType.Divide); var binaryOp = plusParser.Or(minusParser).Or(multiplyParser).Or(divideParser); ``` -------------------------------- ### Handling Tokenization Errors Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Demonstrates how to check for and report tokenization errors. Errors include no recognizer match, zero-width tokens, and overflow. ```csharp var result = tokenizer.TryTokenize("123 + "); if (!result.HasValue) { Console.WriteLine($"Error: {result.ErrorMessage}"); Console.WriteLine($"At: {result.ErrorPosition}"); } ``` -------------------------------- ### Create Token List Parser Success and Failure Results Source: https://github.com/datalust/superpower/blob/dev/_autodocs/02-result-types.md Shows how to construct results for token stream parsing. Use TokenListParserResult.Value for successful parses and TokenListParserResult.Empty for failures, specifying expectations. ```csharp TokenListParserResult.Value(parsedValue, location, remainder) TokenListParserResult.Empty(input, expectations) ``` -------------------------------- ### TryTokenize Method Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Safely converts an input string into a token list without throwing exceptions. Returns a Result object that indicates success or failure. ```APIDOC ## TryTokenize (Public) ### Description Convert input to token list without throwing. ### Method Signature ```csharp public Result> TryTokenize(string source) ``` ### Parameters #### Path Parameters - **source** (string) - Required - Source text to tokenize ### Returns `Result>` — Result with tokens or error ### Example ```csharp var result = tokenizer.TryTokenize("input"); if (result.HasValue) { var tokens = result.Value; // Process tokens } else { Console.WriteLine(result.ErrorMessage); } ``` ### Non-Throwing Safe to use when error handling is needed. ``` -------------------------------- ### TryParse Token List (C#) Source: https://github.com/datalust/superpower/blob/dev/_autodocs/11-parser-extensions.md Parses a token list without throwing exceptions, returning a `TokenListParserResult`. This is the recommended approach for production parsing, allowing for graceful error handling. ```csharp public static TokenListParserResult TryParse( this TokenListParser parser, TokenList input) { ``` ```csharp var result = parser.TryParse(tokenList); if (result.HasValue) { var ast = result.Value; // Process AST } else { Console.WriteLine($"Error: {result.ErrorMessage}"); Console.WriteLine($"At token position: {result.ErrorTokenPosition}"); } ``` -------------------------------- ### Create TextSpan Until Another Span Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Illustrates creating a TextSpan that covers the characters from the beginning of the source up to the position of another specified span. This is useful for extracting prefixes. ```csharp var source = new TextSpan("hello world"); var beforeSpace = source.Until(source.Skip(5)); // beforeSpace covers "hello" ``` -------------------------------- ### Match Specific Keyword Token Source: https://github.com/datalust/superpower/blob/dev/_autodocs/08-token-parsers.md Creates a parser that matches a specific keyword token by its kind and value. Useful for parsing reserved words in a language. ```csharp public enum TokenKind { Keyword, Identifier } // Create parser that matches "if" keyword var ifKeyword = Token.EqualToValue(TokenKind.Keyword, "if"); // Or with case insensitivity var ifKeywordIgnoreCase = Token.EqualToValueIgnoreCase(TokenKind.Keyword, "if"); ``` -------------------------------- ### Non-Throwing Parsing with TryParse Source: https://github.com/datalust/superpower/blob/dev/_autodocs/12-exceptions.md Recommended for avoiding exceptions and improving control flow. Use TryParse to handle parsing results without relying on exceptions. ```csharp var result = parser.TryParse(sourceCode); if (result.HasValue) { var ast = result.Value; // Process AST } else { // Error: result.ErrorMessage, result.ErrorPosition ReportError(result); } ``` -------------------------------- ### Parse Token List (C#) Source: https://github.com/datalust/superpower/blob/dev/_autodocs/11-parser-extensions.md Parses a token list and throws a `ParseException` if parsing fails. Use this when you expect the input to be valid and want to fail fast on errors. ```csharp public static T Parse( this TokenListParser parser, TokenList input) { ``` ```csharp public enum TokenKind { Number, Plus, Minus } var numberToken = Token.EqualTo(TokenKind.Number).Apply(Numerics.IntegerInt32); var result = numberToken.Parse(tokenList); try { int value = result; Console.WriteLine($"Number: {value}"); } catch (ParseException ex) { Console.WriteLine($"Parse error: {ex.Message}"); } ``` -------------------------------- ### Tokenize Method Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Converts an input string into a list of tokens. This method will throw a ParseException if tokenization fails. ```APIDOC ## Tokenize (Public) ### Description Convert input string to token list, throwing on error. ### Method Signature ```csharp public TokenList Tokenize(string source) ``` ### Parameters #### Path Parameters - **source** (string) - Required - Source text to tokenize ### Returns `TokenList` — List of tokens ### Throws `ParseException` — If tokenization fails ### Example ```csharp public class MyTokenizer : Tokenizer { protected override IEnumerable> Tokenize(TextSpan span) { // Implementation } } var tokenizer = new MyTokenizer(); var tokens = tokenizer.Tokenize("input text"); ``` ``` -------------------------------- ### Match Exact String Source: https://github.com/datalust/superpower/blob/dev/_autodocs/07-span-and-text-parsers.md Use this parser to match a specific, case-sensitive string. It returns a TextSpan containing the matched text. ```csharp public static TextParser EqualTo(string text) ``` ```csharp var arrow = Span.EqualTo("=>"); arrow.Parse("=> result"); // TextSpan for "=>" var keyword = Span.EqualTo("while"); keyword.Parse("while (x)"); // TextSpan for "while" ``` -------------------------------- ### Parser Composition for Identifiers Source: https://github.com/datalust/superpower/blob/dev/_autodocs/00-index.md Shows how to compose parsers to create a more complex parser, specifically for identifying valid identifiers in code. ```csharp // Parser composition var identifier = Character.Letter.Then(first => Character.LetterOrDigit.Many() .Select(rest => first + new string(rest))); ``` -------------------------------- ### Combine Parsers with LINQ Source: https://github.com/datalust/superpower/blob/dev/_autodocs/07-span-and-text-parsers.md Shows how to combine multiple TextSpan parsers using LINQ syntax for sequential parsing. This is useful for structured text formats. ```csharp var result = from keyword in Span.EqualTo("while") from ws in Span.Whitespace from condition in Span.AnyCharExcept(')') select (keyword.ToStringValue(), condition.ToStringValue()); ``` -------------------------------- ### Combine Parsers with Or Source: https://github.com/datalust/superpower/blob/dev/_autodocs/03-combinators.md Use the `Or` combinator to try a primary parser and fall back to an alternative if the primary fails. This is useful for matching different but related patterns. ```csharp public static TextParser Or( this TextParser parser, TextParser alternative) ``` ```csharp // Match either a positive or negative number var signedNumber = Character.EqualTo('+').IgnoreThen(Numerics.NaturalUInt32) .Or(Character.EqualTo('-').IgnoreThen(Numerics.NaturalUInt32)); // Match keyword or identifier var keywordOrId = Token.EqualToValue(Kind.Identifier, "if").Or(someIdentifier); ``` -------------------------------- ### Match Any Identifier Token Source: https://github.com/datalust/superpower/blob/dev/_autodocs/08-token-parsers.md Creates a parser that matches any token of the Identifier kind. This is a basic step before extracting the identifier's name or performing other operations. ```csharp public enum TokenKind { Identifier } // Match any Identifier token var anyIdentifier = Token.EqualTo(TokenKind.Identifier); // Extract the identifier name var identifierName = anyIdentifier .Select(t => t.Span.ToStringValue()); ``` -------------------------------- ### Sequence Source: https://github.com/datalust/superpower/blob/dev/_autodocs/08-token-parsers.md Matches multiple tokens in a specific order. ```APIDOC ## Sequence ### Description Match multiple tokens in order. ### Method Signature ```csharp public static TokenListParser[]> Sequence( params TKind[] kinds) ``` ### Parameters #### Path Parameters - **kinds** (TKind[]) - Required - Token kinds to match in sequence ### Returns `TokenListParser[]>` — Array of matched tokens ### Example ```csharp public enum Token { OpenParen, Number, Comma, Number, CloseParen } var sequence = Token.Sequence(Token.OpenParen, Token.Number, Token.Comma, Token.Number, Token.CloseParen); // Matches exactly this sequence: ( num , num ) ``` ### Behavior Tokens must match in exact order. ``` -------------------------------- ### Text Parsing Without Exceptions Source: https://github.com/datalust/superpower/blob/dev/_autodocs/00-index.md Demonstrates how to parse text and handle potential errors using a result type, avoiding exceptions. ```csharp // Text parsing without exceptions var result = numberParser.TryParse("42"); if (result.HasValue) ProcessNumber(result.Value); else ReportError(result.ErrorMessage); ``` -------------------------------- ### Parse Class Methods Source: https://github.com/datalust/superpower/blob/dev/_autodocs/14-all-types.md Provides static methods for creating basic parsers, including chaining, lookaheads, references, fixed values, and sequences. ```APIDOC ## Parse Class ### Description Provides static methods for creating basic parsers. ### Methods - `Chain()` — Left-associative operators - `ChainRight()` — Right-associative operators - `Chain()` — Token version of Chain - `ChainRight()` — Token version of ChainRight - `Not()` — Negative lookahead - `Not()` — Token version of Not - `Ref()` — Lazy recursive reference - `Ref()` — Token version of Ref - `Return()` — Fixed value parser - `Return()` — Token version of Return - `Sequence()` through `Sequence()` — Multiple text parsers - `Sequence()` through `Sequence()` — Multiple token parsers ``` -------------------------------- ### Testing Valid Input Parsing Source: https://github.com/datalust/superpower/blob/dev/_autodocs/13-quick-reference.md Use `TryParse` to validate input and `Assert.True` to confirm a successful parse. Compare the parsed value with the expected result. ```csharp [Fact] public void ParsesValidInput() { var result = parser.TryParse("input"); Assert.True(result.HasValue); Assert.Equal(expected, result.Value); } ``` -------------------------------- ### Match a sequence of token kinds Source: https://github.com/datalust/superpower/blob/dev/_autodocs/08-token-parsers.md Use `Token.Sequence` to match multiple tokens in a specific order. The tokens must appear in the exact sequence provided. ```csharp public enum Token { OpenParen, Number, Comma, Number, CloseParen } var sequence = Token.Sequence(Token.OpenParen, Token.Number, Token.Comma, Token.Number, Token.CloseParen); // Matches exactly this sequence: ( num , num ) ``` -------------------------------- ### Handling Alternative Parsers with .Or() Source: https://github.com/datalust/superpower/blob/dev/_autodocs/12-exceptions.md Use the .Or() combinator to attempt parsing with one parser and fall back to another if the first fails. This is useful for handling expected alternatives in your parsing logic. ```csharp // Try parsing as A, fall back to B on error var aOrB = parserA.Or(parserB); ``` -------------------------------- ### Sequence Source: https://github.com/datalust/superpower/blob/dev/_autodocs/04-parse-helpers.md Apply multiple parsers in order and return a tuple of their results. Overloads support 2, 3, 4, and 5 parsers. ```APIDOC ## Sequence (Multiple Parsers) Apply multiple parsers in order and return a tuple of their results. ### Method Signature (2 parsers) ```csharp public static TextParser<(T, U)> Sequence( TextParser parser1, TextParser parser2) ``` ### Method Signature (3 parsers) ```csharp public static TextParser<(T, U, V)> Sequence( TextParser parser1, TextParser parser2, TextParser parser3) ``` ### Method Signature (4 parsers) ```csharp public static TextParser<(T, U, V, W)> Sequence( TextParser parser1, TextParser parser2, TextParser parser3, TextParser parser4) ``` ### Method Signature (5 parsers) ```csharp public static TextParser<(T, U, V, W, X)> Sequence( TextParser parser1, TextParser parser2, TextParser parser3, TextParser parser4, TextParser parser5) ``` Token versions exist with `TKind` parameter: `Sequence`, etc. **Returns:** Tuple containing results from all parsers **Example:** ```csharp // Parse three separate elements and get all three var (first, second, third) = Parse.Sequence( Character.Letter, Character.Digit, Character.Letter ).Parse("a1b"); // Equivalent to: from c1 in Character.Letter from c2 in Character.Digit from c3 in Character.Letter select (c1, c2, c3) ``` ``` -------------------------------- ### Throwing Pattern for Parsing Source: https://github.com/datalust/superpower/blob/dev/_autodocs/02-result-types.md Demonstrates the throwing pattern for parsing, which is suitable when exceptions are acceptable for error handling. It uses a try-catch block to manage ParseException. ```csharp try { var value = parser.Parse(input); // Use value } catch (ParseException ex) { Console.WriteLine(ex.Message); Console.WriteLine($"At: {ex.ErrorPosition}"); } ``` -------------------------------- ### EqualToValueIgnoreCase Source: https://github.com/datalust/superpower/blob/dev/_autodocs/08-token-parsers.md Matches a token by its kind and its text value, ignoring case during the comparison. The original token text is preserved. ```APIDOC ## EqualToValueIgnoreCase ### Description Matches a token by its kind and its text value, ignoring case during the comparison. The original token text is preserved. ### Method Signature ```csharp public static TokenListParser> EqualToValueIgnoreCase( TKind kind, string value) ``` ### Parameters #### Path Parameters - **kind** (TKind) - Required - Token kind to match - **value** (string) - Required - Text value to match (case-insensitive) ### Returns `TokenListParser>` — Case-insensitive value matcher ### Example ```csharp public enum Keyword { Select } var select = Token.EqualToValueIgnoreCase(Keyword.Select, "select"); // Matches "select", "SELECT", "Select", etc. ``` ### Note Comparison ignores case but returns the original token text. ``` -------------------------------- ### Sequence Parsers with Then, IgnoreThen, ThenIgnore Source: https://github.com/datalust/superpower/blob/dev/_autodocs/03-combinators.md Use Then to chain parsers and process results, IgnoreThen to parse and discard the first result, and ThenIgnore to parse and discard the second result. These are useful for building complex parsing logic step-by-step. ```csharp public static TextParser Then( this TextParser parser, Func> next) public static TextParser IgnoreThen( this TextParser parser, TextParser next) public static TextParser ThenIgnore( this TextParser parser, TextParser next) ``` ```csharp // Parse "a:" where we want just the letter var parser = Character.Letter.ThenIgnore(Character.EqualTo(':')); parser.Parse("a:"); // Returns 'a' // Parse "a:b" and return both letters var ab = Character.Letter.Then(x => Character.EqualTo(':').IgnoreThen( Character.Letter.Select(y => (x, y)) ) ); ab.Parse("a:b"); // Returns ('a', 'b') ``` -------------------------------- ### Match Any Character with AnyChar Parser Source: https://github.com/datalust/superpower/blob/dev/_autodocs/05-character-parsers.md Use `AnyChar` to match and return any single character. It succeeds on any input character. ```csharp public static TextParser AnyChar { get; } ``` ```csharp var any = Character.AnyChar; any.Parse("x abc"); // Returns 'x' ``` -------------------------------- ### Convert TextSpan to String Source: https://github.com/datalust/superpower/blob/dev/_autodocs/10-core-types.md Shows how to convert a TextSpan into its string representation, effectively extracting the characters covered by the span. ```csharp var span = new TextSpan("hello", Position.Zero, 5); string text = span.ToStringValue(); // "hello" ``` -------------------------------- ### Parse Class Methods Source: https://github.com/datalust/superpower/blob/dev/_autodocs/14-all-types.md Provides static methods for creating parsers, including chaining operators, handling negative lookaheads, recursive references, fixed values, and sequences. ```csharp public static class Parse ``` -------------------------------- ### Implement Parameterless Tokenization Logic Source: https://github.com/datalust/superpower/blob/dev/_autodocs/09-tokenization.md Override this method to implement basic tokenization logic. Yield `Result.Value()` for recognized tokens, `Result.Empty()` for errors, and consume the span. Each token must consume at least one character. ```csharp protected virtual IEnumerable> Tokenize(TextSpan span) { var next = span.ConsumeChar(); while (next.HasValue) { if (char.IsWhiteSpace(next.Value)) { // Skip whitespace next = next.Remainder.ConsumeChar(); } else if (next.Value == '+') { yield return Result.Value(TokenKind.Plus, next); next = next.Remainder.ConsumeChar(); } else { yield return Result.Empty(next.Location); break; } } } ``` -------------------------------- ### Or Source: https://github.com/datalust/superpower/blob/dev/_autodocs/03-combinators.md Combines two parsers, attempting the first and falling back to the second if the first fails. This is useful for choosing between alternative parsing strategies. ```APIDOC ## Or ### Description Choose between alternative parsers. This combinator attempts to apply the primary parser and, if it fails, tries the alternative parser. ### Method Signature ```csharp public static TextParser Or(this TextParser parser, TextParser alternative) ``` ### Parameters #### Path Parameters - **parser** (TextParser) - Required - Primary parser to try - **alternative** (TextParser) - Required - Fallback parser if primary fails ### Response #### Success Response (200) - **parser** (TextParser) - Result of whichever parser succeeds first ### Example ```csharp // Match either a positive or negative number var signedNumber = Character.EqualTo('+').IgnoreThen(Numerics.NaturalUInt32) .Or(Character.EqualTo('-').IgnoreThen(Numerics.NaturalUInt32)); // Match keyword or identifier var keywordOrId = Token.EqualToValue(Kind.Identifier, "if").Or(someIdentifier); ``` ``` -------------------------------- ### TokenizationState Class Source: https://github.com/datalust/superpower/blob/dev/_autodocs/14-all-types.md Represents the state of the tokenization process, primarily holding a reference to the previous token. This is useful for context-aware tokenization or analysis. ```APIDOC ## TokenizationState Class ### Description Represents the state of the tokenization process, primarily holding a reference to the previous token. This is useful for context-aware tokenization or analysis. ### Properties - `Previous` (Token) - The previously processed token. ```