### Basic Location Tracking Example Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/location_tracking.md Use @L and @R to capture the start and end byte positions of a token within a rule. 'start' will be the byte location of the token's beginning, and 'end' will be the byte location immediately after the token's end. ```lalrpop Symbol = { => { // `start` is the byte location of the start of our string // `s` is the string itself // `end` is the byte location of the end } } ``` -------------------------------- ### Tokenization Example Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/001_lexer_gen.md Demonstrates how a lexer would tokenize a string containing parentheses and numbers, highlighting the resulting terminals. ```text ( 22 44 ) ) ^ ^^ ^^ ^ ^ | | | | ")" terminal | | | | | | | ")" terminal | +----+ | | | 2 r"[0-9]+" terminals | "(" terminal ``` -------------------------------- ### Install clog-cli for Versioning Source: https://github.com/lalrpop/lalrpop/blob/master/CONTRIBUTING.md Command to install the `clog-cli` tool, which is a prerequisite for the `version.sh` script used in releasing LALRPOP. ```sh cargo install clog-cli ``` -------------------------------- ### Lexer Pseudocode Example Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/001_lexer_gen.md Illustrates the logic of a lexer that checks for specific terminals (string literals and regex) in a loop. ```pseudocode let mut i = 0; // index into string loop { skip whitespace; // we do this implicitly, at least by default if (data at index i is "(") { produce "("; } else if (data at index i is ")") { produce ")"; } else if (data at index i matches regex "[0-9]+") { produce r"[0-9]+"; } } ``` -------------------------------- ### LALRPOP <> shorthand expansion examples Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/003_type_inference.md Illustrates how the <> shorthand expands in various contexts, including single and multiple matched values, and with struct constructors. Useful for understanding implicit naming and value selection. ```lalrpop A => bar(<>) ``` ```lalrpop => bar(a) ``` ```lalrpop A B => bar(<>) ``` ```lalrpop => bar(a, b) ``` ```lalrpop A B => (<>) ``` ```lalrpop => (a, b) ``` ```lalrpop B => bar(<>) ``` ```lalrpop B => bar(a) ``` ```lalrpop B => bar(<>) ``` ```lalrpop B => bar(p) ``` ```lalrpop => bar(<>) ``` ```lalrpop => bar(a, b) ``` ```lalrpop => bar(<>) ``` ```lalrpop => bar(p, q) ``` ```lalrpop B => Foo {<>} ``` ```lalrpop B => Foo {p:p} ``` ```lalrpop => Foo {<>} ``` ```lalrpop => Foo {p:p, q:q} ``` ```lalrpop => format!("<>") ``` ```lalrpop => format!(" ") ``` ```lalrpop => format!("<>") ``` ```lalrpop => format!("

") ``` -------------------------------- ### Location Tracking for Rust Range Construction Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/location_tracking.md Utilize @L and @R to directly construct a Rust Range by binding the start and end byte positions of a token. The resulting range is inclusive of the start and exclusive of the end. ```lalrpop Symbol: Range = { => { start..end } } ``` -------------------------------- ### LALRPOP Grammar Example Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/001_lexer_gen.md An example LALRPOP grammar defining 'Term' and 'Num' nonterminals, including string literals and regular expressions as terminals. ```lalrpop pub Term: i32 = { => n, "(" ")" => t, }; Num: i32 = => i32::from_str(s).unwrap(); ``` -------------------------------- ### LALRPOP Calculator Grammar (calculator4.lalrpop) Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/006_macros.md This grammar defines a simple calculator with basic arithmetic operations. It's a foundational example for LALRPOP usage. ```lalrpop grammar; use ::{ // ... other imports }; // ... grammar rules for calculator4.lalrpop ``` -------------------------------- ### Generate Parser Module with lalrpop_mod! Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/quick_start_guide.md Use this macro in your `lib.rs` or other module files to create a Rust module for your generated parser. For example, `grammar.lalrpop` will create a `grammar` submodule. ```rust lalrpop_mod!(grammar); ``` -------------------------------- ### Define Grammar G0 Source: https://github.com/lalrpop/lalrpop/blob/master/lalrpop/src/lr1/lane_table/README.md This is the first example grammar, G0, used to illustrate LR(1) construction. It is a reduced version of G1 from the Pager and Chen paper. ```lalrpop G0 = X "c" | Y "d" X = "e" X | "e" Y = "e" Y | "e" ``` -------------------------------- ### Tuple Macro with Quantifier in Lalrpop Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/cheatsheet.md Applies a quantifier to a group of matches within parentheses. This example shows a non-terminal that matches a number followed by a comma, repeated zero or more times. ```lalrpop ",")*> ``` -------------------------------- ### Lalrpop Lane Table Example Source: https://github.com/lalrpop/lalrpop/blob/master/lalrpop/src/lr1/lane_table/README.md This table summarizes the lookahead information for conflicting actions (C0, C1, C2) originating from states S0 and S1. It shows which states contribute lookahead and the specific lookahead terminals. ```plaintext | State | C0 | C1 | C2 | Successors | | S0 | | ["c"] | ["d"] | {S1} | | S1 | ["e"] | [] | [] | {S1} | ``` -------------------------------- ### Rust Test for Calculator with Match Declaration Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/001_lexer_gen.md Unit tests in Rust to verify the behavior of a parser after applying a `match` declaration for token precedence. This example demonstrates expected and failing cases. ```rust #[test] fn calculator2b() { // These will all work: let result = calculator2b::TermParser::new().parse("33").unwrap(); assert_eq!(result, "33"); let result = calculator2b::TermParser::new().parse("foo33").unwrap(); assert_eq!(result, "Id(foo33)"); let result = calculator2b::TermParser::new().parse("(foo33)").unwrap(); assert_eq!(result, "Id(foo33)"); // This one will fail: let result = calculator2b::TermParser::new().parse("(22)").unwrap(); assert_eq!(result, "Twenty-two!"); } ``` -------------------------------- ### Define a 'Term' grammar rule in LALRPOP Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/crash_course.md This grammar rule defines a 'Term' which can be either a 'Num' or a parenthesized 'Term'. It serves as a basic example of grammar structure. ```lalrpop Term = Num | "(" Term ")" ``` -------------------------------- ### Capture Position in Lalrpop Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/cheatsheet.md Captures the byte offset of the first and last characters of a matched token. Use `@L` for the start offset and `@R` for the end offset plus one. ```lalrpop T ``` -------------------------------- ### Initialize a new project Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/001_adding_lalrpop.md Create a new binary project using the cargo CLI. ```console cargo new --bin calculator ``` -------------------------------- ### Parse Expression with Lexer and Input Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/004_token_references.md Demonstrates how to create a lexer from an input string, and then use it with the generated LALRPOP parser to parse the expression. The input string is passed to the parser to manage lifetimes. ```rust let input = "22 * pi + 66"; let lexer = Lexer::new(input); let expr = calculator9::ExprParser::new() .parse(input,lexer) .unwrap(); assert_eq!(&format!("{:?}", expr), "((\"22\" * \"pi\") + \"66\")"); ``` -------------------------------- ### Create build.rs script Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/001_adding_lalrpop.md Define the build script to process LALRPOP files in the root directory. ```rust fn main() { lalrpop::process_root().unwrap(); } ``` -------------------------------- ### Configure Build Script with process_src Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/quick_start_guide.md Add this to your `build.rs` file to automatically process LALRPOP grammar files. This function uses default settings to find and compile `.lalrpop` files in the `src/` directory. ```rust fn main() { lalrpop::process_src().unwrap(); } ``` -------------------------------- ### Run Lalrpop Benchmarks Source: https://github.com/lalrpop/lalrpop/blob/master/CONTRIBUTING.md Execute benchmarks for the lalrpop-test package using cargo bench. This command is useful for performance testing and will generate a baseline on the first run. ```shell cargo bench --package lalrpop-test ``` -------------------------------- ### Build and Test LALRPOP with Cargo Source: https://github.com/lalrpop/lalrpop/blob/master/CONTRIBUTING.md Standard commands for building and testing the LALRPOP crate using Cargo. These are used when grammar changes are not involved. ```sh # building $ cargo build # -p lalrpop --release # testing using cargo test $ cargo test # -p lalrpop --release # testing using cargo nextest $ cargo nextest run ``` -------------------------------- ### Iterating Over Beachhead States Source: https://github.com/lalrpop/lalrpop/blob/master/lalrpop/src/lr1/lane_table/README.md Initiates a Depth First Search (DFS) through the state table, starting from 'beachhead' states. These states are identified as those not reachable from other states within the table. ```rust for beachhead in beachheads { ... } ``` -------------------------------- ### Modify Grammar for Error Recovery Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/008_error_recovery.md Update the grammar line to accept a mutable vector for error storage. This setup is required for LALRPOP's error recovery mechanism. ```lalrpop use lalrpop_util::ErrorRecovery; grammar<'err>(errors: &'err mut Vec, &'static str>>); ``` -------------------------------- ### Manually Run LALRPOP Executable Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/quick_start_guide.md Execute the `lalrpop` binary directly from your terminal to generate parser files. This command generates `file.rs` from `file.lalrpop` if the source is newer than the output. ```console lalrpop file.lalrpop ``` -------------------------------- ### Configure LALRPOP to Generate in Source Tree Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/generate_in_source.md Use this configuration to maintain the previous behavior of generating parser files in the same directory as the input files. This is useful for projects that rely on this file placement. ```rust fn main() { lalrpop::Configuration::new() .generate_in_source_tree() .process().unwrap(); } ``` -------------------------------- ### LALR(1) State Representation Source: https://github.com/lalrpop/lalrpop/blob/master/lalrpop/src/lr1/lane_table/README.md Visual representation of an LALR(1) state, showing items and their associated lookahead sets. This example demonstrates how lookaheads are incorporated into state representation. ```text S1 = X = (*) "e" [_] | X = "e" (*) ["c"] // lookahead from C1 | X = (*) "e" "X" [_] | X = "e" (*) X [_] | Y = (*) "e" [_] | Y = "e" (*) ["d"] // lookahead from C2 | Y = (*) "e" Y [_] | Y = "e" (*) Y [_] ``` -------------------------------- ### Apply Default LALRPOP Configuration Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/advanced_setup.md This is the default build script configuration for LALRPOP. It processes .lalrpop files in-place and only regenerates if the file has changed. ```rust fn main() { lalrpop::process_src().unwrap(); } ``` -------------------------------- ### Tuple pattern matching for unwrapping Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/003_type_inference.md Demonstrates using pattern matching with tuples in action code to unwrap nested structures. Useful for handling complex tuple types returned by matched nonterminals. ```lalrpop pub FooBar: String = { <(a, b):Foo> => format!(a, b), <(c, (a, b)):Bar> => format!(c, a, b) } ``` -------------------------------- ### Define LALRPOP Extern Block Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/003_writing_custom_lexer.md Start by defining an `extern` block in your grammar file to expose the parser's API. This block is where you'll declare custom types and tokens. ```lalrpop extern { // ... } ``` -------------------------------- ### Add LALRPOP Dependencies to Cargo.toml Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/quick_start_guide.md Include these lines in your Cargo.toml to add LALRPOP as a build-time and runtime dependency. Ensure versions of `lalrpop` and `lalrpop-util` are synchronized. ```toml # The generated code depends on lalrpop-util. [dependencies] lalrpop-util = "0.23.1" # Add a build-time dependency on the lalrpop library: [build-dependencies] lalrpop = "0.23.1" # If you are supplying your own external lexer you can disable default features so that the # built-in lexer feature is not included # [dependencies] # lalrpop-util = { version = "0.23.1", default-features = false } # # [build-dependencies] # lalrpop = { version = "0.23.1", default-features = false } ``` -------------------------------- ### Run LALRPOP Parser in Rust Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/005_external_lib.md Executes the generated LALRPOP parser by reading source code, creating a lexer, parsing the input, and printing the resulting Abstract Syntax Tree (AST). ```rust let source_code = std::fs::read_to_string("myscript.toy")?; let lexer = Lexer::new(&source_code); let parser = ScriptParser::new(); let ast = parser.parse(lexer)?; println!("{:?}", ast); ``` -------------------------------- ### Run Code Coverage Report with Build Script Inclusion Source: https://github.com/lalrpop/lalrpop/blob/master/CONTRIBUTING.md Use this command to generate an HTML code coverage report that includes build scripts. This provides a more accurate coverage picture, especially for build processes. ```shell cargo llvm-cov --include-build-script report --html ``` -------------------------------- ### Implement Custom Error Logic in LALRPOP Action Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/007_fallible_actions.md Modify LALRPOP grammar actions using `=>?` to return custom errors. This example shows how to map parsing failures to specific variants of a custom error enum, like `InputTooBig` or `OddNumber`. ```lalrpop Num: i32 = { r"[0-9]+" =>? i32::from_str(<>) .map_err(|_| ParseError::User { error: Calculator6Error::InputTooBig }) .and_then(|i| if i % 2 == 0 { Ok(i) } else { Err(ParseError::User { error: Calculator6Error::OddNumber }) }) }; ``` -------------------------------- ### Initial LALRPOP Grammar for Variables and Literals Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/002_raw_delimited_content.md This grammar attempts to define variables and string literals. It fails due to ambiguity between the regex for variables and the regex for string content. ```lalrpop use super::{Var, Lit, Eql}; grammar; pub Var: Var = => <>.chars().next().unwrap().into(); pub Lit: Lit = "\"" "\"" => <>.into(); pub Eql: Eql = "=" => (<>).into(); ``` -------------------------------- ### Build LALRPOP for Grammar Changes Source: https://github.com/lalrpop/lalrpop/blob/master/CONTRIBUTING.md Command to build the LALRPOP crate when changes affect its own grammar. This is a prerequisite for generating the updated parser. ```sh $ cargo build -p lalrpop # --release ``` -------------------------------- ### Configure Grammar Generation Strategy Source: https://github.com/lalrpop/lalrpop/blob/master/RELEASES.md Use these annotations in the grammar declaration to specify the code generation scheme. ```rust #[recursive_ascent] grammar; ``` ```rust #[LALR] grammar; ``` -------------------------------- ### Include generated parser Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/001_adding_lalrpop.md Use the lalrpop_mod! macro to include the generated parser module. ```rust lalrpop_mod!(grammar); ``` -------------------------------- ### Version and Publish LALRPOP Source: https://github.com/lalrpop/lalrpop/blob/master/CONTRIBUTING.md Steps for releasing LALRPOP without using Cargo-Release. This involves updating the version, committing, publishing, and tagging. ```sh Run ./version.sh . Commit the changes Run cargo publish for lalrpop and lalrpop-util Tag new release Push new tag to repository ``` -------------------------------- ### Test LALRPOP grammar with various inputs Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/006_macros.md Provides unit tests for the LALRPOP grammar, verifying parsing of empty input, expressions with and without trailing commas, and multiple comma-separated elements. ```rust use lalrpop_util::lalrpop_mod; lalrpop_mod!(pub calculator5); #[test] fn calculator5() { let expr = calculator5::ExprsParser::new().parse("").unwrap(); assert_eq!(&format!("{:?}", expr), "[]"); let expr = calculator5::ExprsParser::new() .parse("22 * 44 + 66") .unwrap(); assert_eq!(&format!("{:?}", expr), "[((22 * 44) + 66)]"); let expr = calculator5::ExprsParser::new() .parse("22 * 44 + 66,") .unwrap(); assert_eq!(&format!("{:?}", expr), "[((22 * 44) + 66)]"); let expr = calculator5::ExprsParser::new() .parse("22 * 44 + 66, 13*3") .unwrap(); assert_eq!(&format!("{:?}", expr), "[((22 * 44) + 66), (13 * 3)]"); let expr = calculator5::ExprsParser::new() .parse("22 * 44 + 66, 13*3,") .unwrap(); assert_eq!(&format!("{:?}", expr), "[((22 * 44) + 66), (13 * 3)]"); } ``` -------------------------------- ### LALRPOP Calculator Grammar with Error Handling (calculator5.lalrpop) Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/006_macros.md This grammar extends the basic calculator by incorporating error handling mechanisms. It demonstrates how to manage syntax errors within LALRPOP. ```lalrpop grammar; use ::{ // ... other imports }; // ... grammar rules for calculator5.lalrpop ``` -------------------------------- ### Lexer Implementation with Input Reference Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/004_token_references.md Implements a lexer that holds a reference to the input string. This is necessary to create tokens containing slices of the original input. ```rust use std::str::CharIndices; pub struct Lexer<'input> { chars: std::iter::Peekable>, input: &'input str, } impl<'input> Lexer<'input> { pub fn new(input: &'input str) -> Self { Lexer { chars: input.char_indices().peekable(), input, } } } ``` -------------------------------- ### Grammar Parameters in Lalrpop Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/cheatsheet.md Allows input parameters to be used within the generated parser. Define parameters in the grammar declaration, like `grammar(scale: isize);`. ```lalrpop grammar(scale: isize); ``` -------------------------------- ### Define Token Precedence with Match Declaration Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/001_lexer_gen.md Use `match` declarations to explicitly set precedence between terminals. Higher precedence items should come first. The `_` symbol includes all other terminals not explicitly listed. ```lalrpop match { r"[0-9]+" } else { r"\w+", _ } ``` -------------------------------- ### Force Color Output in LALRPOP Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/advanced_setup.md Use this configuration to force colorized output from LALRPOP, regardless of TTY settings. This is useful for consistent output across different environments. ```rust fn main() { lalrpop::Configuration::new() .always_use_colors() .process_current_dir(); } ``` -------------------------------- ### Generalized LALRPOP Grammar with Space in String Literals Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/002_raw_delimited_content.md This grammar attempts to generalize string literals to include spaces. Combined with the `match` declaration, this leads to new parsing errors, demonstrating the limitations of the previous workarounds. ```lalrpop match { r"[x-z]" } else { r"[a-z ]*", _ } pub Var: Var = => <>.chars().next().unwrap().into(); pub Lit: Lit = { "\"" "\"" => <>.into(), "\"" "\"" => <>.into(), }; pub Eql: Eql = "=" => (<>).into(); ``` -------------------------------- ### Referencing All Parameters in Lalrpop Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/cheatsheet.md Uses `<>` to refer to all parameters of a non-terminal as a single tuple. This is useful for type inference. ```lalrpop <> ``` -------------------------------- ### LALRPOP Parser Method Signature Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/002_paren_numbers.md Illustrates the signature of the 'parse' method generated by LALRPOP for a public nonterminal, showing input and potential error types. ```rust fn parse<'input>(&self, input: &'input str) -> Result> // ~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // | | ``` -------------------------------- ### Create Lexer Constructor Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/005_external_lib.md Provides a constructor for the `Lexer` struct, initializing the token stream using the `Token::lexer()` method provided by the Logos trait. ```rust impl<'input> Lexer<'input> { pub fn new(input: &'input str) -> Self { Self { token_stream: Token::lexer(input).spanned() } } } ``` -------------------------------- ### Rust Code to Test LALRPOP Parser Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/002_paren_numbers.md Uses the generated 'TermParser' from the LALRPOP grammar to parse various string inputs. Requires 'lalrpop-util' dependency. ```rust use lalrpop_util::lalrpop_mod; lalrpop_mod!(pub calculator1); // synthesized by LALRPOP #[test] fn calculator1() { assert!(calculator1::TermParser::new().parse("22").is_ok()); assert!(calculator1::TermParser::new().parse("(22)").is_ok()); assert!(calculator1::TermParser::new().parse("((((22))))").is_ok()); assert!(calculator1::TermParser::new().parse("((22)").is_err()); } ``` -------------------------------- ### Define Tokens and Lexical Errors with Logos Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/005_external_lib.md Sets up the Token enum for lexing using the Logos crate, including keywords, identifiers, integers, operators, and custom error handling for invalid tokens or integers. ```rust use std::fmt; use std::num::ParseIntError; use logos::Logos; #[derive(Default, Debug, Clone, PartialEq)] pub enum LexicalError { InvalidInteger(ParseIntError), #[default] InvalidToken, } impl From for LexicalError { fn from(err: ParseIntError) -> Self { LexicalError::InvalidInteger(err) } } #[derive(Logos, Clone, Debug, PartialEq)] #[logos(skip r"[ \t\n\f]+", skip r"#.*\n?", error = LexicalError)] pub enum Token { #[token("var")] KeywordVar, #[token("print")] KeywordPrint, #[regex("[_a-zA-Z][_0-9a-zA-Z]*", |lex| lex.slice().to_string())] Identifier(String), #[regex("[1-9][0-9]*", |lex| lex.slice().parse())] Integer(i64), #[token("(")] LParen, #[token(")")] RParen, #[token("=")] Assign, #[token(";")] Semicolon, #[token("+")] OperatorAdd, #[token("-")] OperatorSub, #[token("*")] OperatorMul, #[token("/")] OperatorDiv, } ``` -------------------------------- ### LALRPOP Grammar with Match Declaration for Lexing Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/002_raw_delimited_content.md This version introduces a `match` declaration to prioritize tokenization. While it resolves some ambiguities, it can lead to incorrect parsing for specific inputs like `z = "x"`. ```lalrpop use super::{Var, Lit, Eql}; grammar; match { r"[x-z]" } else { r"[a-z]*", _ } pub Var: Var = => <>.chars().next().unwrap().into(); pub Lit: Lit = "\"" "\"" => <>.into(); pub Eql: Eql = "=" => (<>).into(); ``` -------------------------------- ### Rust Tests for String Literals Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/002_raw_delimited_content.md A suite of Rust tests demonstrating the correct parsing of variables, string literals, and equality expressions using the defined Lalrpop grammar. ```rust #[test] fn homerun() { assert_eq!(nobol5::VarParser::new().parse("x"), Ok('x'.into())); assert_eq!(nobol5::LitParser::new().parse(r#""abc""#), Ok("abc".into())); assert_eq!(nobol5::EqlParser::new().parse(r#"x = \"a\""#), Ok((\'x\', \"a\").into())); assert_eq!(nobol5::EqlParser::new().parse(r#"y = \"bc\""#), Ok((\'y\', \"bc\").into())); assert_eq!(nobol5::EqlParser::new().parse(r#"z = \"xyz\""#), Ok((\'z\', \"xyz\").into())); assert_eq!(nobol5::EqlParser::new().parse(r#"z = \"x\""#), Ok((\'z\', \"x\").into())); assert_eq!(nobol5::EqlParser::new().parse(r#"z = \"x y z\""#), Ok((\'z\', \"x y z\").into())); } ``` -------------------------------- ### Precedence and Associativity with Attribute Macros Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/004_full_expressions.md Rewrites the tiered expression grammar using `precedence` and `assoc` attribute macros for reduced code complexity. Specifies operator precedence levels and associativity. ```lalrpop pub Expr: i32 = { #[precedence(level="0")] // Highest precedence Term, #[precedence(level="1")] #[assoc(side="left")] "*" => l * r, "/" => l / r, #[precedence(level="2")] #[assoc(side="left")] "+" => l + r, "-" => l - r, }; ``` -------------------------------- ### Implement Error Recovery in Production Rule Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/008_error_recovery.md Add a third alternative to the 'Term' production rule that uses the '!' token. This alternative captures 'ErrorRecovery' values and returns 'Expr::Error'. ```lalrpop Term: Box = { Num => Box::new(Expr::Number(<>)), "(" ")", ! => { errors.push(<>); Box::new(Expr::Error) }, }; ``` -------------------------------- ### Call Parser with State Parameter Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/009_state_parameter.md Instantiate the parser and invoke its `parse` method, passing the state parameter along with the input string. The state parameter must implement the `Copy` trait or be passed as a reference. ```rust let scale = 2; let expr = calculator8::ExprParser::new() .parse(scale,"11 * 22 + 33") .unwrap(); assert_eq!(&format!("{:?}", expr), "((22 * 44) + 66)"); ``` -------------------------------- ### Define Grammar with State Parameter Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/009_state_parameter.md Declare a state parameter for your grammar. This parameter will be passed to the parser during initialization. ```rust grammar(scale: i32); ``` -------------------------------- ### Configure Token::Error in Grammar Extern Block Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/006_error_recovery_custom_lexer.md Inform Lalrpop about the new `Token::Error` variant by adding it to the `extern` block in the `grammar.lalrpop` file. ```diff extern { type Location = usize; type Error = LexicalError; enum Token { + "error" => Token::Error, ``` -------------------------------- ### Define AST for Toy Language Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/005_external_lib.md Defines the Abstract Syntax Tree (AST) structures for statements and expressions, including integers, variables, and binary operations. ```rust #[derive(Clone, Debug, PartialEq)] pub enum Statement { Variable { name: String, value: Box }, Print { value: Box }, } #[derive(Clone, Debug, PartialEq)] pub enum Expression { Integer(i64), Variable(String), BinaryOperation { lhs: Box, operator: Operator, rhs: Box, }, } #[derive(Clone, Debug, PartialEq)] pub enum Operator { Add, Sub, Mul, Div, } ``` -------------------------------- ### Update LALRPOP Generated Parser Source: https://github.com/lalrpop/lalrpop/blob/master/CONTRIBUTING.md Script to regenerate the LALRPOP parser file (`lrgrammar.rs`) after grammar modifications. It backs up the existing file before replacement. ```sh $ sh update_lrgrammar.sh ``` -------------------------------- ### Test Expression Parsing and Formatting Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/005_building_asts.md Write a Rust test function to parse an expression string using the generated LALRPOP parser and assert its formatted Debug output. ```rust lalrpop_mod!(pub calculator4); pub mod ast; #[test] fn calculator4() { let expr = calculator4::ExprParser::new() .parse("22 * 44 + 66") .unwrap(); assert_eq!(&format!("{:?}", expr), "((22 * 44) + 66)"); } ``` -------------------------------- ### Declare Terminals with LALRPOP Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/003_writing_custom_lexer.md Declare each terminal by mapping a parser-visible name (identifier or string literal) to a `lexer::Tok::Variant`. This step is crucial for the parser to interpret lexer tokens. ```lalrpop extern { type Location = usize; type Error = lexer::LexicalError; enum lexer::Tok { " " => lexer::Tok::Space, "\t" => lexer::Tok::Tab, "\n" => lexer::Tok::Linefeed, } } ``` -------------------------------- ### LALRPOP grammar with type and code execution Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/crash_course.md This LALRPOP grammar defines a 'Term' nonterminal with an 'i32' type. It includes code snippets to be executed for matching 'Num' and parenthesized 'Term' rules, demonstrating data conversion. ```lalrpop Term: i32 = { Num => /* ... number code ... */, "(" Term ")" => /* ... parenthesized code ... */, }; ``` -------------------------------- ### LR(0) State Machine for G0 Source: https://github.com/lalrpop/lalrpop/blob/master/lalrpop/src/lr1/lane_table/README.md The LR(0) states for grammar G0 are listed, showing the production rules and the position of the dot. ```text S0 = G0 = (*) X "c" | G0 = (*) Y "d" | X = (*) "e" X | X = (*) "e" | Y = (*) "e" Y | Y = (*) "e" S1 = X = "e" (*) X | X = "e" (*) | X = (*) "e" | X = (*) "e" "X" | Y = "e" (*) Y | Y = "e" (*) | Y = (*) "e" | Y = (*) "e" Y S2 = X = "e" X (*) S3 = G0 = X (*) "c" S4 = Y = "e" Y (*) S5 = G0 = Y (*) "d" S6 = G0 = X "c" (*) S7 = G0 = Y "d" (*) ``` -------------------------------- ### Using Lexer Input in Rust Parser Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/003_writing_custom_lexer.md Configure your Rust parser to accept a `Lexer` instance as input instead of a string slice. This change is enabled by the LALRPOP `extern` block configuration. ```rust let lexer = lexer::Lexer::new("\n\n\n"); match parser::parse_Program(lexer) { ... } ``` -------------------------------- ### Lexer Tokenization Logic Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/004_token_references.md The `next` method of the lexer iterates through characters, skipping whitespace, recognizing parentheses and operators, and capturing sequences of other characters as `NumSymbol` tokens, storing them as slices of the original input. ```rust impl<'input> Iterator for Lexer<'input> { type Item = Spanned, usize, ()>; fn next(&mut self) -> Option { loop { match self.chars.next() { Some((_, ' ')) | Some((_, '\n')) | Some((_, '\t')) => continue, Some((i, ')')) => return Some(Ok((i, Tok::ParenClose, i + 1))), Some((i, '(')) => return Some(Ok((i, Tok::ParenOpen, i + 1))), Some((i, '+')) => return Some(Ok((i, Tok::ExprOp(Opcode::Add), i + 1))), Some((i, '-')) => return Some(Ok((i, Tok::ExprOp(Opcode::Sub), i + 1))), Some((i, '*')) => return Some(Ok((i, Tok::FactorOp(Opcode::Mul), i + 1))), Some((i, '/')) => return Some(Ok((i, Tok::FactorOp(Opcode::Div), i + 1))), None => return None, // End of file Some((i,_)) => { loop { match self.chars.peek() { Some((j, ')'))|Some((j, '('))|Some((j, '+'))|Some((j, '-'))|Some((j, '*'))|Some((j, '/'))|Some((j,' ')) => return Some(Ok((i, Tok::NumSymbol(&self.input[i..*j]), *j))), None => return Some(Ok((i, Tok::NumSymbol(&self.input[i..]),self.input.len()))), _ => { self.chars.next(); } } } } } } } } ``` -------------------------------- ### LALRPOP Grammar for Expression AST Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/005_building_asts.md Define LALRPOP grammar rules to parse expressions and build `Expr` AST nodes. Imports are necessary for custom types like `Expr` and `Opcode`. ```lalrpop use std::str::FromStr; use ast::{Expr, Opcode}; // (0) grammar; pub Expr: Box = { // (1) Expr ExprOp Factor => Box::new(Expr::Op(<>)), // (2) Factor, }; ExprOp: Opcode = { // (3) "+" => Opcode::Add, "-" => Opcode::Sub, }; ``` -------------------------------- ### Implement Display for Token Enum Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/005_external_lib.md Implements the `Display` trait for the `Token` enum, which is required by LALRPOP for including tokens in error messages. ```rust impl fmt::Display for Token { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "{:?}", self) } } ``` -------------------------------- ### Return Result from LALRPOP Action Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/007_fallible_actions.md Use `=>?` instead of `=>` in LALRPOP grammar rules to enable action code to return a `Result`. This is useful when direct value production is not feasible or when integrating with Rust's error handling. ```lalrpop Num: i32 = { r"[0-9]+" =>? i32::from_str(<>) .map_err(|_| ParseError::User { error: "number is too big" }) }; ``` -------------------------------- ### Define LALRPOP Grammar Rules Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/005_external_lib.md Constructs the grammar rules for a simple programming language, defining the structure for scripts, statements (variable assignment and print), and expressions with operator precedence. ```lalrpop pub Script: Vec = { => stmts } pub Statement: ast::Statement = { "var" "=" ";" => { ast::Statement::Variable { name, value } }, "print" ";" => { ast::Statement::Print { value } }, } pub Expression: Box = { #[precedence(level="1")] Term, #[precedence(level="2")] #[assoc(side="left")] "*" => { Box::new(ast::Expression::BinaryOperation { lhs, operator: ast::Operator::Mul, rhs }) }, "/" => { Box::new(ast::Expression::BinaryOperation { lhs, operator: ast::Operator::Div, rhs }) }, #[precedence(level="3")] #[assoc(side="left")] "+" => { Box::new(ast::Expression::BinaryOperation { lhs, operator: ast::Operator::Add, rhs }) }, "-" => { Box::new(ast::Expression::BinaryOperation { lhs, operator: ast::Operator::Sub, rhs }) }, } pub Term: Box = { => { Box::new(ast::Expression::Integer(val)) }, => { Box::new(ast::Expression::Variable(name)) }, "(" ")", } ``` -------------------------------- ### Test Error Recovery with Multiple Errors Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/008_error_recovery.md This test demonstrates parsing strings with various syntax errors. The parser recovers using the '!' token, allowing multiple errors to be detected and reported. ```rust #[test] fn calculator7() { let mut errors = Vec::new(); let expr = calculator7::ExprsParser::new() .parse(&mut errors, "22 * + 3") .unwrap(); assert_eq!(&format!("{:?}", expr), "[((22 * error) + 3)]"); let expr = calculator7::ExprsParser::new() .parse(&mut errors, "22 * 44 + 66, *3") .unwrap(); assert_eq!(&format!("{:?}", expr), "[((22 * 44) + 66), (error * 3)]"); let expr = calculator7::ExprsParser::new() .parse(&mut errors, "*") .unwrap(); assert_eq!(&format!("{:?}", expr), "[(error * error)]"); assert_eq!(errors.len(), 4); } ``` -------------------------------- ### Num definition with <> shorthand Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/003_type_inference.md This definition uses the <> shorthand to synthesize a name for the matched regular expression and use it in the action code. Useful for single-match expressions where the matched value is directly used. ```lalrpop Num: i32 = r"[0-9]+" => i32::from_str(<>).unwrap(); ``` -------------------------------- ### LALRPOP Grammar with Explicit Handling for String Content Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/002_raw_delimited_content.md This grammar attempts to fix the string literal parsing by explicitly defining cases for `[x-z]` and `[a-z]*` within string literals. This is a fragile workaround. ```lalrpop pub Lit: Lit = { "\"" "\"" => <>.into(), "\"" "\"" => <>.into(), }; ``` -------------------------------- ### Original String Literal Rule Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/002_raw_delimited_content.md This rule implicitly defines separate tokens for string delimiters and content, leading to potential tokenization issues. ```lalrpop pub Lit: Lit = "\"" "\"" => <>.into(); ``` -------------------------------- ### Rust Unit Tests for LALRPOP Precedence Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/001_lexer_gen.md These unit tests verify the behavior of the LALRPOP grammar, demonstrating that the fixed string '22' is correctly parsed as 'Twenty-two!' while '222' is parsed as a regular number. ```rust #[test] fn calculator2b() { let result = calculator2b::TermParser::new().parse("33").unwrap(); assert_eq!(result, "33"); let result = calculator2b::TermParser::new().parse("(22)").unwrap(); assert_eq!(result, "Twenty-two!"); let result = calculator2b::TermParser::new().parse("(222)").unwrap(); assert_eq!(result, "222"); } ``` -------------------------------- ### Tiered Expression Grammar for Calculator Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/tutorial/004_full_expressions.md Defines a grammar for arithmetic expressions using tiers to enforce operator precedence. Each tier (Expr, Factor, Term) handles a different level of operations. ```lalrpop use std::str::FromStr; grammar; pub Expr: i32 = { "+" => l + r, "-" => l - r, Factor, }; Factor: i32 = { "*" => l * r, "/" => l / r, Term, }; Term: i32 = { Num, "(" ")", }; Num: i32 = { r"[0-9]+" => i32::from_str(<>).unwrap(), }; ``` -------------------------------- ### Unwrapping Tuple Parameters in Lalrpop Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/cheatsheet.md Unwraps the tuple return of a non-terminal into individual parameters. The syntax `<(a, b, c):Foo>` assigns elements of the tuple `Foo` to `a`, `b`, and `c`. ```lalrpop <(a, b, c):Foo> ``` -------------------------------- ### LALRPOP Grammar Rules with String Literals Source: https://github.com/lalrpop/lalrpop/blob/master/doc/src/lexer_tutorial/003_writing_custom_lexer.md Grammar rules can now use string literals directly, which LALRPOP automatically substitutes with the corresponding `Tok` enum variants defined in the `extern` block. This allows existing rules to work without modification. ```lalrpop FlowCtrl: ast::Stmt = { " " " "