### LLLPG start rule and top-level declaration rules Source: http://ecsharp.net/lllpg/lllpg-part-2 Shows a start rule that parses zero or more using statements followed by top-level declarations. LLLPG does not require a single start rule; any non-private rule can be a starting point. ```LLLPG rule Start @[ UsingStmt* TopLevelDecl* ]; rule TopLevelDecl @[ NamespaceStmt | TypeDecl ]; ... ``` -------------------------------- ### WhateverB Lexer Example Source: http://ecsharp.net/lllpg/lllpg-part-4 Demonstrates the use of Skip and MatchAny in a lexer rule. ```csharp // Example LLLPG(lexer) { rule WhateverB @[ (_|EOF) [b='B']? ]; } // Generated code void WhateverB() { int la0; Skip(); la0 = LA0; if (la0 == 'B') b = MatchAny(); } ``` -------------------------------- ### LLLPG start rule and top-level declaration Source: http://ecsharp.net/lllpg/3-parsing-terminology Shows a start rule that parses zero or more using statements followed by top-level declarations, and a top-level declaration rule that can be a namespace or type declaration. ```LLLPG rule Start @{ UsingStmt* TopLevelDecl* }; rule TopLevelDecl @{ NamespaceStmt | TypeDecl }; ... ``` -------------------------------- ### Block-call statement examples: get/set and general form Source: http://ecsharp.net/lllpg/lemp-processing-model.html EC# block-call statements: get/set are never keywords; instead they are examples of block-call statements. The general form is 'identifier { statements; }' or 'identifier (argument, list) { statements; }'. ```ecsharp get { return _foo; } set { _foo = value; } ``` ```ecsharp identifier { statements; } identifier (argument, list) { statements; } ``` -------------------------------- ### TryMatchExcept Example and Generated Code Source: http://ecsharp.net/lllpg/lllpg-part-4 Example of using TryMatchExcept in an LLLPG parser rule and the corresponding generated C# code. ```csharp // Example (remember that _ does NOT match EOF) LLLPG(parser) { [recognizer { bool TryMatchExcept(); }] rule MatchExcept @[ _ ~A ~(A|B) ~(A|B|C) ~(A|B|C|D) ~(A|B|C|D|E) (~E | EOF) ]; } // Generated code static readonly HashSet MatchExcept_set0 = NewSet(A, B, C, D, E, EOF); static readonly HashSet MatchExcept_set1 = NewSet(E); void MatchExcept() { /* Omitted for brevity */ } bool Try_TryMatchExcept(int lookaheadAmt) { using (new SavePosition(this, lookaheadAmt)) return TryMatchExcept(); } bool TryMatchExcept() { if (!TryMatchExcept()) return false; if (!TryMatchExcept(A)) return false; if (!TryMatchExcept(A, B)) return false; if (!TryMatchExcept(A, B, C)) return false; if (!TryMatchExcept(A, B, C, D)) return false; if (!TryMatchExcept(MatchExcept_set0)) return false; if (!TryMatchExcept(MatchExcept_set1)) return false; return true; } ``` -------------------------------- ### NewSet and NewSetOfRanges Usage Example Source: http://ecsharp.net/lllpg/lllpg-part-4 Demonstrates how LLLPG generates code using NewSet and NewSetOfRanges for rule matching. ```csharp // Example: LLLPG(lexer) { rule Vowel @[ 'a'|'e'|'i'|'o'|'u'|'A'|'E'|'I'|'O'|'U' ]; rule MaybeHexDigit @[ ['0'..'9'|'a'..'f'|'A'..'F']? ]; }; // Generated code: static readonly HashSet Vowel_set0 = NewSet( 'A', 'E', 'I', 'O', 'U', 'a', 'e', 'i', 'o', 'u'); void Vowel() { Match(Vowel_set0); } static readonly HashSet MaybeHexDigit_set0 = NewSetOfRanges('0', '9', 'A', 'F', 'a', 'f'); void MaybeHexDigit() { int la0; la0 = LA0; if (MaybeHexDigit_set0.Contains(la0)) Skip(); } ``` -------------------------------- ### FiveEvenDigits Lexer Example Source: http://ecsharp.net/lllpg/lllpg-part-4 Shows how Match is used to validate specific characters and sets in a lexer rule. ```csharp // Example LLLPG(lexer) { rule FiveEvenDigits @[ zero:='0' ('0'|'2') ('0'|'2'|'4') ('0'|'2'|'4'|'6') ('0'|'2'|'4'|'6'|'8') ]; } // Generated code static readonly HashSet FiveEvenDigits_set0 = NewSet('0', '2', '4', '6', '8'); void FiveEvenDigits() { var zero = Match('0'); Match('0', '2'); Match('0', '2', '4'); Match('0', '2', '4', '6'); Match(FiveEvenDigits_set0); } ``` -------------------------------- ### TryMatchRange Example and Generated Code Source: http://ecsharp.net/lllpg/lllpg-part-4 Example of using TryMatchRange within an LLLPG lexer rule and the resulting generated C# code. ```csharp // Example LLLPG(lexer) { [recognizer { bool ScanLetterDigit(); }] rule LetterDigit @[ ('a'..'z'|'A'..'Z') '0'..'9' ]; } // Generated code void LetterDigit() { MatchRange('A', 'Z', 'a', 'z'); MatchRange('0', '9'); } bool Try_ScanLetterDigit(int lookaheadAmt) { using (new SavePosition(this, lookaheadAmt)) return ScanLetterDigit(); } bool ScanLetterDigit() { if (!TryMatchRange('A', 'Z', 'a', 'z')) return false; if (!TryMatchRange('0', '9')) return false; return true; } ``` -------------------------------- ### LetterDigit Lexer Example Source: http://ecsharp.net/lllpg/lllpg-part-4 Demonstrates the use of MatchRange to validate character ranges. ```csharp // Example LLLPG(lexer) { rule LetterDigit @[ ('a'..'z'|'A'..'Z') '0'..'9' ]; } // Generated code void LetterDigit() { MatchRange('A', 'Z', 'a', 'z'); MatchRange('0', '9'); } ``` -------------------------------- ### Consonant or vowel rule Source: http://ecsharp.net/lllpg/lllpg-part-3 Example of a rule with multiple alternatives and a default branch. ```csharp LLLPG(lexer) { rule ConsonantOrNot @{ ('A'|'E'|'I'|'O'|'U') {Vowel();} / 'A'..'Z' {Consonant();} }; } void ConsonantOrNot() { switch (LA0) { // (Newlines between cases removed for brevity) case 'A': case 'E': case 'I': case 'O': case 'U': { Skip(); Vowel(); } break; default: { MatchRange('A', 'Z'); Consonant(); } break; } } ``` -------------------------------- ### Conceptual MLCommentLine with explicit exit priority Source: http://ecsharp.net/lllpg/lllpg-part-4 A conceptual example demonstrating a desired syntax where the exit branch priority is explicitly managed between other branches. ```LES public token MLCommentLine(ref nested::int)::bool @[ ( &{nested>0} "*/" {nested--;} / "/*" {nested++;} / exit / ~(' '|' ') )* (Newline {return false;} | "*/" {return true;}) ]; ``` -------------------------------- ### Required using directives and StringToken struct Source: http://ecsharp.net/lllpg/lllpg-part-5 Required preamble for the Expr example. Add this above ExprParser and reference Loyc.Syntax.dll, Loyc.Collections.dll, and Loyc.Essentials.dll to compile. ```C# using System; using System.Linq; using System.Collections.Generic; using System.Diagnostics; using Loyc; using Loyc.Syntax; using Loyc.Syntax.Lexing; struct StringToken : ISimpleToken { public string Type { get; set; } public object Value { get { return Type; } } public int StartIndex { get; set; } } ``` -------------------------------- ### LLLPG command-line options (--help output) Source: http://ecsharp.net/lllpg/lllpg-configuration.html Lists the command-line options supported by LLLPG, as reported by LLLPG -help. These options are rarely necessary but include --forcelang, --help, --inlang, --macros, --max-expand, --noparallel, --nostdmacros, --outext, --outlang, --parallel, --set, --snippet, --timeout, and --verbose. Note that in Visual Studio, the [Verbosity(N)] grammar attribute does not work without the --verbose option. ```text --forcelang: Specifies that --inlang overrides the input file extension. Without this option, known file extensions override --inlang. --help: show this screen --inlang=name: Set input language: --inlang=ecs for Enhanced C#, --inlang=les for LES --macros=filename.dll: load macros from given assembly --max-expand=N: stop expanding macros after N nested or iterated expansions. --noparallel: Process all files in sequence --nostdmacros: Don't scan LeMP.StdMacros.dll or pre-import LeMP and LeMP.Prelude --outext=name: Set output extension and optional suffix: .ecs (Enhanced C#), .cs (C#), .les (LES) This can include a suffix before the extension, e.g. --outext=.output.cs If --outlang is not used, output language is chosen by file extension. --outlang=name: Set output language independently of file extension --parallel: Process all files in parallel (this is the default) --set:key=literal: Associate a value with a key (use #get(key) to read it back) --snippet:key=code: Associate code with a key (use #get(key) to read it back) --timeout=N: Aborts the processing thread(s) after this many seconds (0=never) --verbose: Print extra status messages (e.g. discovered Types, list output files). ``` -------------------------------- ### EC# block-call statements for get and set Source: http://ecsharp.net/lllpg/lllpg-part-1 Shows block-call statements for get and set accessors in EC#. In EC#, get and set are never keywords; these are examples of block-call statements, which are equivalent to method calls with a block argument. ```EC# get { return _foo; } set { _foo = value; } ``` -------------------------------- ### LLLPG lexer with token rules Source: http://ecsharp.net/lllpg/8-managing-ambiguity.html A complete lexer grammar using `token` for top-level token rules and `rule` for the Start rule. The `token` modifier avoids ambiguity warnings by replacing the follow set with `_*`. The example includes a comment about the placement of `@{...}` blocks. ```LLLPG public struct Tok { ... } LLLPG(lexer) { public rule List Start @{ {var ts = new List();} ts+=Token* EOF {return ts;} }; token Tok Token() { // BTW: Instead of writing a @{...} block with {...} actions inside, // LLLPG lets you write a {...} block with @{...} blocks inside. But // currently the @{...} blocks must be at the top level of the method, // not nested inside anything else such as a try {...} region. Token t; @{ t=Spaces | t=Id | t=Int | t=Op }; return t; } token Tok Spaces @{ (' '|'\t')+ {return new Token(...);} }; token Tok Id @{ ('a'..'z'|'A'..'Z') ('a'..'z'|'A'..'Z'|'0'..'9')+ {return new Token(...);} }; token Tok Int @{ ('0'..'9')+ {return new Token(...);} }; token Tok Op @{ op:=('+'|'-'|'*'|'/'|'=')+ {return new Token(...);} }; } ``` -------------------------------- ### Nested if statement input example Source: http://ecsharp.net/lllpg/lllpg-part-4 An example of input that creates ambiguity regarding which if statement an else clause belongs to. ```text if (expr) if (expr) Stmt(); else Stmt(); ``` -------------------------------- ### Equivalent @init and inline action rules Source: http://ecsharp.net/lllpg/lllpg-part-1 Shows that the @init clause is supported but not needed; the two rules are equivalent. The first uses @init to run an action before parsing, the second uses an inline semantic action. ```csharp rule1 @init { Action(); }: X | Y; rule2 : { Action(); } (X | Y); ``` -------------------------------- ### Nonsensical gate example Source: http://ecsharp.net/lllpg/lllpg-part-2 Example of a gate producing nonsensical code: the prediction checks for 'A' but the matching consumes 'Q'. This is shown as a warning not to use gates in this way. ```C# // 'A' => 'Q' la0 = LA0; if (la0 == 'A') Match('Q'); ``` -------------------------------- ### Manual LES parsing pipeline with TokensToTree Source: http://ecsharp.net/lllpg/lllpg-part-5 Shows how to manually construct the LES parsing pipeline with a token tree. `TokensToTree` converts the lexer output to a tree, and `LesParser` consumes it. ```csharp var input = (UString)"{ w = (x + y) * z >> (-1); };"; var errOut = new ConsoleMessageSink(); var lexer = new LesLexer(input, "", errOut); var tree = new TokensToTree(lexer, true); // <= Convert tokens to tree! var parser = new LesParser(tree.Buffered(), lexer.SourceFile, errOut); var results = parser.ParseStmtsLazy().Buffered(); ``` -------------------------------- ### LA0, LA(int i), EOF Source: http://ecsharp.net/lllpg/lllpg-api-reference.html Properties and constants for lookahead access. LA0 gets the current lookahead token, LA(i) gets the lookahead at offset i, and EOF is a constant representing end of input. ```APIDOC ## LA0, LA(int i), EOF ### Description Provides access to lookahead tokens. LA0 returns the current lookahead, LA(i) returns the lookahead at offset i, and EOF is a constant for end of input. ### Method Property / Method / Constant ### Endpoint N/A (SDK method) ### Parameters #### Parameters - **i** (int) - Required - The lookahead offset. ### Return Value - **LaType** - The lookahead value. ### Example ``` // Usage in generated code int la0 = LA0; LaType la1 = LA(1); if (LA0 == EOF) { ... } ``` ``` -------------------------------- ### Keyword parsing with LLLPG (initial attempt) Source: http://ecsharp.net/lllpg/lllpg-part-5 Initial keyword parsing example with a rule `IdOrKeyword` that matches keywords and identifiers, and a `ScanNextToken` rule that uses it. Uses `[k(9)]` to increase lookahead to 9, but has two problems: `foreach` is unreachable (matched as `for` + `each`) and words like "form" or "ifone" are parsed as keyword followed by identifier. ```LLLPG [k(9)] private token TT IdOrKeyword @[ "do" {return TT.Do; } / "if" {return TT.If; } / "for" {return TT.For; } / "foreach" {return TT.Foreach; } / "while" {return TT.While; } / "function" {return TT.Function;} / Identifier {return TT.Id; } ]; public token ScanNextToken() @[ Spaces { return TT.Spaces; } / t:IdOrKeyword / t:Operator / t:Literal / ... { return t; } ]; ``` -------------------------------- ### Equivalent expansion of 'any..in' Source: http://ecsharp.net/lllpg/4-lllpg-grammar-features Shows the equivalent explicit alternation for the 'any..in' version, listing each word rule with its own capture. ```LLLPG rule int SumWords @{ (x+:One ' ' / x+:Two ' ' / x+:Ten ' ')* {return x.Sum();} }; ``` -------------------------------- ### C# example of ambiguous method overloading Source: http://ecsharp.net/lllpg/8-managing-ambiguity.html Shows a C# class hierarchy with two overloaded methods that create ambiguity in meaning, though not in parsing. The example is used to illustrate ambiguity that is not a parsing issue. ```csharp class A { public virtual void F(Foo x) {...} } class B : A { public override void F(Foo x) {...} // first public void F(Bar x) {...} // second } ``` -------------------------------- ### Long identifier input example Source: http://ecsharp.net/lllpg/lllpg-part-2 Example input showing why character-level parsing cannot use finite lookahead: the parser must read all 31 characters of the identifier before seeing the '=' that distinguishes an assignment from a call. ```plaintext this_name_is_31_characters_long = 42; ``` -------------------------------- ### MatchExceptRange Usage and Generated Code Source: http://ecsharp.net/lllpg/lllpg-part-4 Shows how to use MatchExceptRange in a lexer rule and the resulting generated code. ```csharp // Example LLLPG(lexer) { rule NotInRanges @[ ~('0'..'9') ~('a'..'z'|'A'..'Z') ]; } // Generated code void NotInRanges() { MatchExceptRange('0', '9'); MatchExceptRange('A', 'Z', 'a', 'z'); } ``` -------------------------------- ### Ambiguity without Suppression Source: http://ecsharp.net/lllpg/lllpg-part-4 An example where no warnings are suppressed. ```LLLPG token CompareOp() @[ ">=" | "<=" / '>' | '<' | '=' ]; ``` -------------------------------- ### Token rule and generated code with char.IsLetter ordering Source: http://ecsharp.net/lllpg/lllpg-part-2 Shows the `Token` rule and its generated method. The generated code examines LA0 before checking `&{char.IsLetter(...)}`, which is more efficient than calling the predicate on every input character. This example is used to argue that testing character values before zero-width assertions is generally good. ```EC# [FullLLk] rule Token @[ Spaces / Word / Number / Punctuation / Comma / _ ]; ``` ```C# void Token() { int la0, la1; la0 = LA0; switch (la0) { case 't': case ' ': Spaces(); break; case '.': { la1 = LA(1); if (la1 >= '0' && la1 <= '9') Number(); else Punctuation(); } break; case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': Number(); break; case '!': case '#': case '$': case '%': case '&': case '*': case '+': case ',': case '-': case '/': case '<': case '=': case '>': case '?': case '@': case '\': case '^': case '|': Punctuation(); break; default: if (la0 >= 'A' && la0 <= 'Z' || la0 >= 'a' && la0 <= 'z') Word(); else if (la0 >= 128 && la0 <= 65532) { la0 = LA0; if (char.IsLetter((char) la0)) Word(); else MatchExcept(); } else MatchExcept(); break; } } ``` -------------------------------- ### ANTLR-like syntax mode example Source: http://ecsharp.net/lllpg/lllpg-in-antlr-style.html Shows the structure of an LLLPG block in ANTLR mode, with general options, code generation options, and ANTLR-style rules. The block can only contain rules, not ordinary C# methods or fields. ```csharp class MyParser : BaseParserForList { [/* general options (as usual) */] LLLPG (/* code generation options (as usual) */) @{ /* ANTLR-style rules (that's the new part) */ myRule [ArgType argument] returns [ReturnType result] : grammar ; }; } ``` -------------------------------- ### Ambiguous nested if input for greedy example Source: http://ecsharp.net/lllpg/8-managing-ambiguity.html Shows the ambiguous input that triggers the greedy/nongreedy decision: an if statement nested inside another if statement, where the else clause could pair with either the inner or outer if. This example illustrates why the follow set of IfStmt includes TT.Else Stmt. ```LLLPG if (expr) if (expr) Stmt(); else Stmt(); ``` -------------------------------- ### LLLPG.exe command-line options Source: http://ecsharp.net/lllpg/lllpg-configuration.html LLLPG can be invoked from the command line or in a pre-build step by running LLLPG.exe with a filename. The following options are available. ```APIDOC ## LLLPG.exe command-line options ### Description LLLPG can be invoked from the command line or in a pre-build step by running LLLPG.exe with a filename. The following options are available. ### Method Command-line invocation ### Endpoint LLLPG.exe _filename_ ### Parameters #### Command-line Options - **--forcelang** (flag) - Optional - Specifies that --inlang overrides the input file extension. Without this option, known file extensions override --inlang. - **--help** (flag) - Optional - Show this screen. - **--inlang=name** (string) - Optional - Set input language: --inlang=ecs for Enhanced C#, --inlang=les for LES. - **--macros=filename.dll** (string) - Optional - Load macros from given assembly. - **--max-expand=N** (integer) - Optional - Stop expanding macros after N nested or iterated expansions. - **--noparallel** (flag) - Optional - Process all files in sequence. - **--nostdmacros** (flag) - Optional - Don't scan LeMP.StdMacros.dll or pre-import LeMP and LeMP.Prelude. - **--outext=name** (string) - Optional - Set output extension and optional suffix: .ecs (Enhanced C#), .cs (C#), .les (LES). This can include a suffix before the extension, e.g. --outext=.output.cs. If --outlang is not used, output language is chosen by file extension. - **--outlang=name** (string) - Optional - Set output language independently of file extension. - **--parallel** (flag) - Optional - Process all files in parallel (this is the default). - **--set:key=literal** (string) - Optional - Associate a value with a key (use #get(key) to read it back). - **--snippet:key=code** (string) - Optional - Associate code with a key (use #get(key) to read it back). - **--timeout=N** (integer) - Optional - Aborts the processing thread(s) after this many seconds (0=never). - **--verbose** (flag) - Optional - Print extra status messages (e.g. discovered Types, list output files). ### Notes - In Visual Studio, the [Verbosity(N)] grammar attribute doesn't work without the --verbose option. ``` -------------------------------- ### Terminal matching rule Source: http://ecsharp.net/lllpg/lllpg-part-3 Example of a simple terminal matching rule in LLLPG. ```text rule PlusMinus @{ '+'|'-' }; ``` -------------------------------- ### MatchExcept Usage and Generated Code Source: http://ecsharp.net/lllpg/lllpg-part-4 Demonstrates how MatchExcept is used in LLLPG rules and the corresponding generated C# code. ```csharp // Example (remember that _ does NOT match EOF) LLLPG(parser) { rule MatchExcept @[ _ ~A ~(A|B) ~(A|B|C) ~(A|B|C|D) ~(A|B|C|D|E) (~E | EOF) ]; } // Generated code static readonly HashSet NotA_set0 = NewSet(A, B, C, D, E, EOF); static readonly HashSet NotA_set1 = NewSet(E); void MatchExcept() { MatchExcept(); MatchExcept(A); MatchExcept(A, B); MatchExcept(A, B, C); MatchExcept(A, B, C, D); MatchExcept(NotA_set0); MatchExcept(NotA_set1); } ``` -------------------------------- ### Optimistic alternative selection Source: http://ecsharp.net/lllpg/lllpg-part-3 Example of an optimistic rule definition where LLLPG assumes no erroneous inputs. ```text rule Either @{ 'A' | B }; ``` -------------------------------- ### Implement ToString for BaseParserForList Source: http://ecsharp.net/lllpg/lllpg-part-4 Example implementation of the required ToString method for BaseParserForList using a TokenType enum. ```csharp protected override string ToString(int tokenType) { return ((TokenType)tokenType).ToString(); } ``` -------------------------------- ### Replace Parser class with grammar for new language Source: http://ecsharp.net/lllpg/6-how-to-write-a-parser.html Provides the complete grammar for the new language, including rules for expressions, primary expressions, and expression lists. Uses actions to build LNode objects with the LNodeFactory. ```csharp partial class Parser { LLLPG (parser(matchType: int, laType: TokenType, terminalType: Token, listInitializer: VList _ = new VList())); alias("(" = TT.LParen); alias(")" = TT.RParen); alias("+" = TT.Add); alias("-" = TT.Sub); alias("," = TT.Comma); rule LNode ExpressionAndEof @{ // Usually you should define a rule that checks for EOF at the end, // otherwise bad input like "5 x" can parse successfully (as a literal 5) result:Expression EOF }; rule LNode Expression @{ result:PrimaryExpr [ // Infix operator op:("+"|"-") PrimaryExpr {$result = F.Call((Symbol) op.Value, $result, $PrimaryExpr);} ]* }; rule LNode PrimaryExpr @{ result:Atom [ // Method call "(" ExpressionList ")" {$result = F.Call($result, $ExpressionList);} ]* }; rule VList ExpressionList @{ result+:Expression ["," result+:Expression]* }; rule LNode Atom @{ t:TT.Number { return F.Literal(t); } | t:TT.Id { return F.Id(t); } | "(" Expression ")" { return F.InParens($Expression); } | error {Error(0, "Expected subexpression");} (_|EOF) { return F.Missing; } }; } ```