pest

https://pest.rs/book/
Admin
pest is a Rust library for writing plain-text parsers using Parsing Expression Grammars (PEGs) that...
Tokens:33,019
Snippets:206
Trust Score:8.3
Update:11 months ago
Show doc for...
Tokens:
Raw
### Pest PEG Grammar Definition Examples

Source: https://pest.rs/book/grammars/peg

Examples of defining basic PEG rules for recognizing numbers and expressions using Pest's grammar syntax. These rules illustrate how to define patterns for sequences of characters and alternatives.

```pest
number = {
    ASCII_DIGIT+
}
expression = {
    number
    | "true"
}
```

--------------------------------

### AWK Clone: Command-Line Execution Examples

Source: https://pest.rs/book/examples/awk

Provides various examples of running the AWK clone using `cargo run` with different arguments, demonstrating basic field processing, regex matching, statistical calculations, CSV parsing, and file-based program execution.

```bash
# Basic field processing
cargo run -- -p '{ print $1, $2 }' employees.txt

# Pattern matching with regex
cargo run -- -p '/Engineer/ { print $1, "is an engineer" }' employees.txt

# Statistical processing
cargo run -- -p 'BEGIN {
                 sum = 0
                 count = 0 
                } 
                $2 > 25 {
                  sum += $2
                  count++ 
                } 
                END { print "Average age over 25:", sum/count }' employees.txt

# CSV processing with custom separator
echo "name,age,dept
Alice,25,Engineering
Bob,30,Sales" | cargo run -- -F "," -p '{ print $1, ": ", $3 }'

# Reading program from file
echo '$2 > 30 { print $1, "is over 30" }' > filter.awk
cargo run -- -f filter.awk employees.txt
```

--------------------------------

### AWK Example Program

Source: https://pest.rs/book/examples/awk

A simple AWK program demonstrating the pattern-action model, including BEGIN, regex matching, field processing, and END blocks.

```awk
BEGIN { print "Processing employee data..." }
/Engineer/ { engineers++ }
$2 > 30 { print $1, "is over 30 years old" }
END { print "Found", engineers, "engineers" }
```

--------------------------------

### Pest PEG Repetition Example

Source: https://pest.rs/book/grammars/peg

Demonstrates the greedy nature of repetition PEG expressions in Pest. The example shows how `ASCII_DIGIT+` consumes as many digits as possible from the input string.

```pest
ASCII_DIGIT+      // one or more characters from '0' to '9'
```

--------------------------------

### Example Program Output

Source: https://pest.rs/book/examples/json

This demonstrates the expected output when running the Rust program after setting up 'data.json' with example JSON content. It shows the program successfully parsing the JSON and then serializing it back into a formatted string, including nested objects, arrays, and escaped characters.

```shell
$ cargo run
  [ ... ]
{"nesting":{"inner object":{}},"an array":[1.5,true,null,0.000001],"string with escaped double quotes":"\"quick brown foxes\""}


```

--------------------------------

### Pest PEG Expression Logic Example

Source: https://pest.rs/book/grammars/peg

A concise representation of the fundamental logic of PEG expressions in Pest. It shows how a sequence of operations is handled, either by success and proceeding, or by failure and trying an alternative.

```pest
(this ~ next_thing) | (other_thing)
```

--------------------------------

### Example J Code Snippets

Source: https://pest.rs/book/examples/jlang

A collection of J language code snippets demonstrating various operations including exponentiation, monadic and dyadic operations, global variable assignment, and string assignments. These examples are intended to be parsed by the Rust program.

```j
_2.5 ^ 3
*: 4.8
title =: 'Spinning at the Boundary'
*: _1 2 _3 4
1 2 3 + 10 20 30
1 + 10 20 30
1 2 3 + 10
2 | 0 1 2 3 4 5 6 7
another =: 'It''s Escaped'
3 | 0 1 2 3 4 5 6 7
(2+1)*(2+2)
3 * 2 + 1
1 + 3 % 4
x =: 100
x - 1
y =: x - 1
y
```

--------------------------------

### Rust Pest Parser Setup for AWK

Source: https://pest.rs/book/examples/awk

Initializes the Pest parser for AWK. This involves deriving the `Parser` trait from a grammar file (`awk.pest`) and importing necessary components from the `pest` and `pest_derive` crates. This setup enables the transformation of raw input into a structured parse tree.

```rust
use pest::Parser;
use pest::iterators::{Pair, Pairs};
use pest::pratt_parser::{Assoc, Op, PrattParser};
use anyhow::{Result, anyhow};
use crate::ast::*;

#[derive(pest_derive::Parser)]
#[grammar = "awk.pest"]
pub struct AwkParser;

```

--------------------------------

### Pest PEG No-Backtracking Example

Source: https://pest.rs/book/grammars/peg

Shows a Pest PEG rule (`ANY* ~ ANY`) that demonstrates the no-backtracking behavior. The `ANY*` greedily consumes the entire input, causing the subsequent `ANY` to fail, and the failure propagates without backtracking.

```pest
word = {     // to recognize a word...
    ANY*
    ~ ANY
}
```

--------------------------------

### Pest PEG Ordered Choice Operator Example

Source: https://pest.rs/book/grammars/peg

Illustrates the usage and importance of the ordered choice operator (`|`) in Pest PEGs. It highlights how the order of alternatives affects parsing, especially when one alternative is a prefix of another.

```pest
"true" | "false"
```

```pest
"a" | "ab"
```

--------------------------------

### Pest: Start (SOI) and End (EOI) of Input Anchors

Source: https://pest.rs/book/grammars/syntax

Demonstrates how to use the SOI (Start of Input) and EOI (End of Input) rules in Pest grammars. These rules match the beginning and end of the input string without consuming characters, ensuring a rule matches the entire input.

```pest
main = {
    SOI
    ~ (...) 
    ~ EOI
}
```

--------------------------------

### Accessing Pest Parse Tokens

Source: https://pest.rs/book/parser_api

Demonstrates how to access and iterate over tokens generated by a successful Pest parse. It shows how to get a token iterator from a parse result and print each token.

```rust
let parse_result = Parser::parse(Rule::sum, "1773 + 1362").unwrap();
let tokens = parse_result.tokens();

for token in tokens {
    println!("{:?}", token);
}
```

--------------------------------

### Handling Rules with Unknown Sub-Rule Counts

Source: https://pest.rs/book/parser_api

Explains that rules with quantifiers like `*` (zero or more) or `+` (one or more) do not have a fixed number of sub-rules, making direct `.next().unwrap()` calls unsafe. The example grammar shows a `number*` rule.

```pest
list = { number* }
```

--------------------------------

### Python: String Interpolation with Expressions

Source: https://pest.rs/book/grammars/syntax

A Python example demonstrating string interpolation using f-strings, where expressions like '2 + 4' can be embedded and evaluated within the string, preserving normal whitespace behavior.

```python
#!/bin/env python3
print(f"The answer is {2 + 4}.")
```

--------------------------------

### Pest: Operator Precedence Example

Source: https://pest.rs/book/grammars/syntax

Shows the precedence of Pest grammar operators. Repetition operators have the highest precedence, followed by predicates, sequence, and ordered choice. This example clarifies how complex rules are parsed by demonstrating equivalent explicit grouping.

```pest
my_rule = {
    "a"* ~ "b"?
    | &"b"+ ~ "a"
}

// equivalent to

my_rule = {
      ( ("a"*) ~ ("b"?) )
    | ( (&("b"+)) ~ "a" )
}
```

--------------------------------

### Rust: Accessing Tagged Nodes in Pest Parse Trees

Source: https://pest.rs/book/grammars/syntax

Provides Rust code examples for interacting with tagged elements in Pest parse trees. It demonstrates iterating through pairs, checking for tags using `as_node_tag`, and finding tagged elements with `find_first_tagged` and `find_tagged`.

```rust
let pairs = ExampleParser::parse(Rule::example_rule, example_input).unwrap();
for pair in pairs.clone() {
    if let Some(tag) = pair.as_node_tag() {
        // ...
    }
}
let first = pairs.find_first_tagged("tag");
let all_tagged = pairs.find_tagged("tag");
```

--------------------------------

### Pest Calculator Grammar Definition

Source: https://pest.rs/book/index

Defines the grammar for a simple calculator using the pest parser generator. It covers number parsing, arithmetic operations, and expression structure. This grammar is used to tokenize and structure mathematical expressions.

```pest
num = @{ int ~ ("-" ~ ASCII_DIGIT*)? ~ ( E"e" ~ int)? }
int = { ("+" | "-")? ~ ASCII_DIGIT+ }

operation = _{ add | subtract | multiply | divide | power }
    add      = { "+" }
    subtract = { "-" }
    multiply = { "*" }
    divide   = { "/" }
    power    = { "^" }

expr = { term ~ (operation ~ term)* }
term = _{ num | "(" ~ expr ~ ")" }

calculation = _{ SOI ~ expr ~ EOI}

WHITESPACE = _{ " " | "\t" }
```

--------------------------------

### Define Whitespace and Equation Rules for Pest Parser

Source: https://pest.rs/book/examples/calculator

This snippet defines the 'WHITESPACE' rule to ignore spaces and the main 'equation' rule. The 'equation' rule ensures that the entire input consists of a valid expression from start ('SOI') to end ('EOI'), preventing partial matches.

```pest
WHITESPACE = _{ " " }

// We can't have SOI and EOI on expr directly, because it is used
// recursively (e.g. with parentheses)
equation = _{ SOI ~ expr ~ EOI }
```

--------------------------------

### Defining Pest Grammar Rules

Source: https://pest.rs/book/grammars/syntax

Demonstrates the basic syntax for defining rules in Pest grammars. Rules are named and can optionally include documentation comments. Special symbols like `_` (silent) and `@` (atomic) can precede the rule definition to modify its behavior. Whitespace is ignored, and comments start with `//`.

```pest
//! Grammar doc
my_rule = { ... }

/// Rule doc
another_rule = {        // comments are preceded by two slashes
    ...                 // whitespace goes anywhere
}
```

```pest
silent_rule = { _ ... }
atomic_rule = { @ ... }
```

--------------------------------

### Iterating Over Inner Rules with Pairs

Source: https://pest.rs/book/parser_api

Shows how to use the `Pairs` iterator, returned by `into_inner()`, to process sub-rules. This example parses a sum expression, extracts numbers, and verifies their values and types.

```rust
let pairs = Parser::parse(Rule::sum, "1773 + 1362")
    .unwrap().next().unwrap()
    .into_inner();

let numbers = pairs
    .clone()
    .map(|pair| str::parse(pair.as_str()).unwrap())
    .collect::<Vec<i32>>();
assert_eq!(vec![1773, 1362], numbers);

for (found, expected) in pairs.zip(vec!["1773", "1362"]) {
    assert_eq!(Rule::number, found.as_rule());
    assert_eq!(expected, found.as_str());
}
```

--------------------------------

### AWK Program Structure (Pest Grammar)

Source: https://pest.rs/book/examples/awk

Defines the top-level structure of an AWK program, including BEGIN, END, and regex patterns, rules (pattern + action), actions, and the complete program structure enclosed by start and end of input markers.

```pest
begin_pattern = { "BEGIN" }
end_pattern = { "END" }
regex_pattern = { regex }
expr_pattern = { expr }

pattern = @{
    begin_pattern |
    end_pattern |
    regex_pattern |
    expr_pattern
}

rule = { (pattern ~ action) | action | pattern }
action = { "{" ~ statement* ~ "}" }

program = { SOI ~ rule* ~ EOI }
```

--------------------------------

### Parse J Expressions in Rust

Source: https://pest.rs/book/examples/jlang

This Rust code snippet defines how to parse different J language constructs into an Abstract Syntax Tree (AST). It handles decimal numbers, expressions, and identifiers. It uses Pest's `pair.as_str()` to get string representations and `parse::<f64>()` for floating-point conversion. Error handling is done via `panic!` for unexpected terms.

```rust
Rule::decimal => {
            let dstr = pair.as_str();
            let (sign, dstr) = match &dstr[..1] {
                "_" => (-1.0, &dstr[1..]),
                _ => (1.0, &dstr[..]),
            };
            let mut flt: f64 = dstr.parse().unwrap();
            if flt != 0.0 {
                // Avoid negative zeroes; only multiply sign by nonzeroes.
                flt *= sign;
            }
            AstNode::DoublePrecisionFloat(flt)
        }
        Rule::expr => build_ast_from_expr(pair),
        Rule::ident => AstNode::Ident(String::from(pair.as_str())),
        unknown_term => panic!("Unexpected term: {:?}", unknown_term),
    }
}
```

--------------------------------

### J Lexical Rules: Integer, Decimal, Identifier, String (Pest)

Source: https://pest.rs/book/examples/jlang

Defines lexing rules for J's integers (allowing leading underscore for negatives), decimals, identifiers (starting with a letter, followed by alphanumerics or underscore), and strings (enclosed in single quotes, with escaped quotes). The `@` modifier makes these rules atomic.

```pest
integer = @{ "_"? ~ ASCII_DIGIT+ }

decimal = @{ "_"? ~ ASCII_DIGIT+ ~ "." ~ ASCII_DIGIT* }

ident = @{ ASCII_ALPHA ~ (ASCII_ALPHANUMERIC | "_")* }

string = @{ "'" ~ ( "''" | (!"'" ~ ANY) )* ~ "'" }
```

--------------------------------

### Cargo Help and Version Commands

Source: https://pest.rs/book/examples/awk

Demonstrates how to use Cargo to automatically generate help messages and display version information for a project. These commands are standard practices for CLI applications to provide user assistance.

```bash
# Automatic help generation
carp run -- --help

# Version information
carp run -- --version

```

--------------------------------

### Pest.rs: Define Top-Level JSON File Rule

Source: https://pest.rs/book/examples/json

Defines the Pest.rs grammar rule for an entire JSON file. It requires the file to start with Start of Input (SOI), contain either an object or an array, and end with End of Input (EOI). This rule is marked silent.

```pest
json = _{ SOI ~ (object | array) ~ EOI }
```

--------------------------------

### Pest Grammar: Match Same Text vs. Same Pattern

Source: https://pest.rs/book/grammars/syntax

Demonstrates the difference between matching the exact same text multiple times using the stack (PUSH, POP) versus matching the same pattern. The PUSH and POP keywords allow for exact text matching, while the standard pattern matching allows for any valid sequence within the pattern.

```pest
same_text = {
    PUSH( "a" | "b" | "c" )
    ~ POP
}
same_pattern = {
    ("a" | "b" | "c")
    ~ ("a" | "b" | "c")
}
```

--------------------------------

### Rust Project Dependencies for AWK Clone

Source: https://pest.rs/book/examples/awk

Lists the necessary Rust crates for building the AWK interpreter, including pest for parsing, clap for CLI arguments, regex for pattern matching, and anyhow for error handling.

```toml
[dependencies]
pest = "2.8"
pest_derive = "2.8"
clap = { version = "4", features = ["derive"] }
regex = "1.5"
anyhow = "1"
lazy_static = "1.4"
```

--------------------------------

### Error Handling with `anyhow`

Source: https://pest.rs/book/examples/awk

Demonstrates context-aware error messages and error chaining using the `anyhow` crate for robust error propagation in Rust applications.

```rust
// Context-aware error messages
Err(anyhow!("Failed to read program file '{}': {}", file, e))

// Error chaining and propagation
parse_program(&program_text)?
```

--------------------------------

### Rust CSV Parser Initialization and Usage

Source: https://pest.rs/book/examples/csv

Initializes a Rust project for a CSV tool using Cargo and sets up dependencies for the pest and pest_derive crates. It then demonstrates basic usage of the generated CSVParser for parsing individual fields.

```bash
$ cargo init --bin csv-tool
     Created binary (application) project
$ cd csv-tool
```

```toml
[dependencies]
pest = "2.6"
pest_derive = "2.6"
```

```rust
use pest::Parser;
use pest_derive::Parser;

#[derive(Parser)]
#[grammar = "csv.pest"]
pub struct CSVParser;

fn main() {
    let successful_parse = CSVParser::parse(Rule::field, "-273.15");
    println!("{:?}", successful_parse);

    let unsuccessful_parse = CSVParser::parse(Rule::field, "this is not a number");
    println!("{:?}", unsuccessful_parse);
}
```

--------------------------------

### Regex Pattern Matching with `regex`

Source: https://pest.rs/book/examples/awk

Shows how to use the `regex` crate for compiling and using regular expressions, including compile-time error checking for pattern validity.

```rust
// Compile-time error checking for regex patterns
let regex = Regex::new(regex_str)?;
```

--------------------------------

### Rust Doc Comments (/// and //! ) in Pest

Source: https://pest.rs/book/grammars/comments

Illustrates Pest's use of Rust-style doc comments. `//!` documents the entire grammar file, while `///` documents specific rules or elements like enums.

```rust
//! A parser for JSON file.

json = { ... }

/// Matches object, e.g.: `{ "foo": "bar" }`
object = { ... }

```

```rust
/// A parser for JSON file.
enum Rule {
    json,
    /// Matches object, e.g.: `{ "foo": "bar" }`
    object,
}

```

--------------------------------

### Rust Project Dependencies for AWK Clone

Source: https://pest.rs/book/examples/awk

Lists the necessary dependencies for the AWK clone project in `Cargo.toml`. Includes `pest` for parsing, `clap` for CLI arguments, `regex` for pattern matching, `anyhow` for error handling, and `lazy_static` for global static variables.

```toml
[package]
name = "awk-clone"
version = "0.1.0"
edition = "2021"

[dependencies]
pest = "2.8"
pest_derive = "2.8"
clap = { version = "4", features = ["derive"] }
regex = "1.5"
anyhow = "1"
lazy_static = "1.4"

```

--------------------------------

### Calculator Main Function with Pest Parsing

Source: https://pest.rs/book/examples/calculator

This Rust `main` function demonstrates the usage of the Pest parser for a calculator application. It reads lines from standard input, parses them as 'equation' rules using `CalculatorParser`, and then uses the `parse_expr` function to convert the parsed pairs into an AST. Errors during parsing are printed to stderr.

```rust
fn main() -> io::Result<()> {
    for line in io::stdin().lock().lines() {
        match CalculatorParser::parse(Rule::equation, &line?) {
            Ok(mut pairs) => {
                println!(
                    "Parsed: {:#?}",
                    parse_expr(
                        // inner of expr
                        pairs.next().unwrap().into_inner()
                    )
                );
            }
            Err(e) => {
                eprintln!("Parse failed: {:?}", e);
            }
        }
    }
    Ok(())
}


```

--------------------------------

### J Whitespace and Comment Lexical Rules (Pest)

Source: https://pest.rs/book/examples/jlang

Defines whitespace in J as solely spaces and tabs, excluding newlines which are significant for statement delimitation. Comments start with 'NB.' and extend to the end of the line, crucially not consuming the trailing newline.

```pest
WHITESPACE = _{ " " | "\t" }

COMMENT = _{ "NB." ~ (!"\n" ~ ANY)* }
```

--------------------------------

### Rust CSV Grammar Definition with Pest

Source: https://pest.rs/book/examples/csv

Defines the grammar for parsing CSV files using the pest parser generator. It specifies rules for fields (numbers), records (fields separated by commas), and the overall file structure including start and end of input markers.

```pest
field = { (ASCII_DIGIT | "." | "-")+} 
record = { field ~ ("," ~ field)* } 
file = { SOI ~ (record ~ ("\r\n" | "\n"))* ~ EOI }
```

--------------------------------

### Pest Calculator Grammar Definition

Source: https://pest.rs/book/intro

Defines the grammar for a simple calculator using Pest's syntax. It includes rules for numbers, operations, expressions, and terms, along with whitespace handling and start/end markers.

```pest
num = @{ int ~ ("\." ~ ASCII_DIGIT*)? ~ (^"e" ~ int)? }
int = { ("+" | "-")? ~ ASCII_DIGIT+ }

operation = _{ add | subtract | multiply | divide | power }
    add      = { "+" }
    subtract = { "-" }
    multiply = { "*" }
    divide   = { "/" }
    power    = { "^" }

expr = { term ~ (operation ~ term)* }
term = _{ num | "(" ~ expr ~ ")" }

calculation = _{ SOI ~ expr ~ EOI }

WHITESPACE = _{ " " | "\t" }
```

--------------------------------

### Rust PrattParser for Calculator Expression

Source: https://pest.rs/book/precedence

Initializes a PrattParser in Rust for parsing calculator expressions based on a Pest grammar. It defines the precedence and associativity of infix, postfix, and prefix operators. This setup is crucial for correctly evaluating expressions with mixed operator priorities.

```rust
let pratt = 
    PrattParser::new()
        .op(Op::infix(Rule::add, Assoc::Left) | Op::infix(Rule::sub, Assoc::Left))
        .op(Op::infix(Rule::mul, Assoc::Left) | Op::infix(Rule::div, Assoc::Left))
        .op(Op::infix(Rule::pow, Assoc::Right))
        .op(Op::postfix(Rule::fac))
        .op(Op::prefix(Rule::neg));
```

--------------------------------

### Parse CSV Content with Pest in Rust

Source: https://pest.rs/book/examples/csv

Invokes the Pest parser to process the CSV file content. It expects a `pest::iterators::Pair` representing the 'file' rule and uses `expect` for error handling during parsing. The result is unwrapped to get the first 'file' rule.

```rust
fn main() {
    // ...

    let file = CSVParser::parse(Rule::file, &unparsed_file)
        .expect("unsuccessful parse") // unwrap the parse result
        .next().unwrap(); // get and unwrap the `file` rule; never fails

    // ...
}
```

--------------------------------

### Main function for J Parser in Rust

Source: https://pest.rs/book/examples/jlang

This Rust `main` function reads a J program from 'example.ijs', parses it using the `parse` function (presumably defined elsewhere using Pest), and prints the resulting AST to the console. It includes basic error handling for file reading and parsing operations.

```rust
fn main() {
    let unparsed_file = std::fs::read_to_string("example.ijs")
      .expect("cannot read ijs file");
    let astnode = parse(&unparsed_file).expect("unsuccessful parse");
    println!("{:?}", &astnode);
}
```

--------------------------------

### Expected AST Output from J Parser

Source: https://pest.rs/book/examples/jlang

This output represents the Abstract Syntax Tree (AST) generated by the Rust parser when run against the provided J code examples. It shows the structured representation of J expressions, including different operation types and their operands.

```text
$ cargo run
  [ ... ]
[Print(DyadicOp { verb: Power, lhs: DoublePrecisionFloat(-2.5),
    rhs: Integer(3) }),
Print(MonadicOp { verb: Square, expr: DoublePrecisionFloat(4.8) }),
Print(IsGlobal { ident: "title", expr: Str("Spinning at the Boundary") }),
Print(MonadicOp { verb: Square, expr: Terms([Integer(-1), Integer(2),
    Integer(-3), Integer(4)]) }),
Print(DyadicOp { verb: Plus, lhs: Terms([Integer(1), Integer(2), Integer(3)]),
    rhs: Terms([Integer(10), Integer(20), Integer(30)]) }),
Print(DyadicOp { verb: Plus, lhs: Integer(1), rhs: Terms([Integer(10),
    Integer(20), Integer(30)]) }),
Print(DyadicOp { verb: Plus, lhs: Terms([Integer(1), Integer(2), Integer(3)]),
    rhs: Integer(10) }),
Print(DyadicOp { verb: Residue, lhs: Integer(2),
    rhs: Terms([Integer(0), Integer(1), Integer(2), Integer(3), Integer(4),
    Integer(5), Integer(6), Integer(7)]) }),
Print(IsGlobal { ident: "another", expr: Str("It\'s Escaped") }),
Print(DyadicOp { verb: Residue, lhs: Integer(3), rhs: Terms([Integer(0),
    Integer(1), Integer(2), Integer(3), Integer(4), Integer(5),
    Integer(6), Integer(7)]) }),
Print(DyadicOp { verb: Times, lhs: DyadicOp { verb: Plus, lhs: Integer(2),
    rhs: Integer(1) }, rhs: DyadicOp { verb: Plus, lhs: Integer(2),
        rhs: Integer(2) } }),
Print(DyadicOp { verb: Times, lhs: Integer(3), rhs: DyadicOp { verb: Plus,
    lhs: Integer(2), rhs: Integer(1) } }),
Print(DyadicOp { verb: Plus, lhs: Integer(1), rhs: DyadicOp { verb: Divide,
    lhs: Integer(3), rhs: Integer(4) } }),
Print(IsGlobal { ident: "x", expr: Integer(100) }),
Print(DyadicOp { verb: Minus, lhs: Ident("x"), rhs: Integer(1) }),
Print(IsGlobal { ident: "y", expr: DyadicOp { verb: Minus, lhs: Ident("x"),
    rhs: Integer(1) } }),
Print(Ident("y"))]

```

--------------------------------

### Pest: Handling Optional Whitespace and Comments

Source: https://pest.rs/book/grammars/syntax

Shows how to define and use optional whitespace and comments in Pest grammars using WHITESPACE and COMMENT rules. These rules can be implicitly inserted between tokens, allowing for more flexible parsing of formats like code.

```pest
expression = { "4" ~ "+" ~ "5" }
WHITESPACE = _{ " " }
COMMENT = _{ "/*" ~ (!"*/" ~ ANY)* ~ "*/" }
```

```pest
expression = {
    "4"   ~ (ws | com)*
    ~ "+" ~ (ws | com)*
    ~ "5"
}
ws = _{ " " }
com = _{ "/*" ~ (!"*/" ~ ANY)* ~ "*/" }
```

```pest
WHITESPACE = _{ " " }
expression = { "4" ~ "+" ~ "5" }
main = { SOI ~ expression ~ EOI }
```

--------------------------------

### Pest Parser Grammar Definition

Source: https://pest.rs/book/parser_api

Defines a simple grammar for the Pest parser, including rules for numbers, enclosed text, and sums. This grammar serves as a basis for demonstrating token generation.

```pest
number = { ASCII_DIGIT+ }                // one or more decimal digits
enclosed = { "(.." ~ number ~ 	"..)" }    // for instance, "(..6472..)"
sum = { number ~ " + " ~ number }        // for instance, "1362 + 12"
```

--------------------------------

### Handle AWK Print Statement in Rust

Source: https://pest.rs/book/examples/awk

This Rust function handles the execution of AWK's `print` statement. It supports printing all fields ($0) if no expressions are provided, or specific evaluated expressions separated by the output field separator. It relies on `eval_expr` to get values and uses `output_record_separator`.

```rust
Statement::Print(exprs) => {
    if exprs.is_empty() {
        // Print $0
        print!("{}{}", self.fields[0], self.output_record_separator);
    } else {
        let values: Result<Vec<_>> = exprs.iter()
            .map(|expr| self.eval_expr(expr))
            .collect();
        let values = values?;
        let output: Vec<String> = values.iter()
            .map(|v| v.to_string())
            .collect();
        print!("{}{}", output.join(&self.output_field_separator), self.output_record_separator);
    }
}
```

--------------------------------

### Rust AWK Clone Main Application Logic

Source: https://pest.rs/book/examples/awk

The main function of the AWK clone application, demonstrating argument parsing with clap, file I/O for program and input, parsing the AWK program using pest, and executing it with a custom interpreter. It handles errors using the anyhow crate for better context.

```rust
use clap::{Arg, Command};
use std::fs;
use std::io::{self, Read};
use anyhow::Result;

// Import our modules
mod ast;
mod parser;
mod interpreter;

use parser::*;
use interpreter::*;

fn main() -> Result<()> {
    let matches = Command::new("awk-clone")
        .version("1.0")
        .author("Your Name <your.email@example.com>")
        .about("A simple AWK clone built with pest")
        .long_about("
This AWK clone demonstrates how to build a complete language interpreter using Rust and pest.
It supports AWK's core features including pattern-action programming, field processing,
built-in variables, and control flow constructs.

Examples:
  awk-clone -p '{ print $1, $2 }' data.txt
  awk-clone -f script.awk input.txt
  cat data.txt | awk-clone -p 'BEGIN { sum = 0 } { sum += $2 } END { print sum }'
        ")
        .arg(Arg::new("program")
            .short('p')
            .long("program")
            .value_name("PROGRAM")
            .help("AWK program string")
            .long_help("AWK program as a command-line string. Cannot be used with --file.")
            .required_unless_present("file")
            .conflicts_with("file"))
        .arg(Arg::new("file")
            .short('f')
            .long("file")
            .value_name("FILE")
            .help("AWK program file")
            .long_help("Read AWK program from a file. Cannot be used with --program.")
            .required_unless_present("program")
            .conflicts_with("program"))
        .arg(Arg::new("input")
            .help("Input file (reads from stdin if not provided)")
            .long_help("Input data file to process. If not provided, reads from standard input.")
            .index(1))
        .arg(Arg::new("field-separator")
            .short('F')
            .long("field-separator")
            .value_name("FS")
            .help("Field separator pattern")
            .long_help("Set the field separator. Default is whitespace. Examples: -F ',' for CSV, -F ':' for /etc/passwd"))
        .get_matches();

    // Parse program source
    let program_text = if let Some(prog) = matches.get_one::<String>("program") {
        prog.clone()
    } else if let Some(file) = matches.get_one::<String>("file") {
        fs::read_to_string(file)
            .map_err(|e| anyhow::anyhow!("Failed to read program file '{}': {}", file, e))? 
    } else {
        unreachable!("clap should ensure either program or file is provided");
    };

    // Read input data
    let input_text = if let Some(input_file) = matches.get_one::<String>("input") {
        fs::read_to_string(input_file)
            .map_err(|e| anyhow::anyhow!("Failed to read input file '{}': {}", input_file, e))? 
    } else {
        let mut buffer = String::new();
        io::stdin().read_to_string(&mut buffer)
            .map_err(|e| anyhow::anyhow!("Failed to read from stdin: {}", e))?;
        buffer
    };

    // Parse the AWK program
    let program = parse_program(&program_text)
        .map_err(|e| anyhow::anyhow!("Parse error: {}", e))?;

    // Create and configure interpreter
    let mut interpreter = Interpreter::new();
    
    if let Some(fs) = matches.get_one::<String>("field-separator") {
        interpreter.set_field_separator(fs.clone());
    }

    // Execute the program
    interpreter.run_program(&program, &input_text)
        .map_err(|e| anyhow::anyhow!("Runtime error: {}", e))?;

    Ok(())
}

```

--------------------------------

### Define Operator Precedence with PrattParser

Source: https://pest.rs/book/examples/calculator

This snippet shows how to define operator precedence for arithmetic operations using Pest's Pratt parser. It utilizes `lazy_static` to create a global instance of `PrattParser`. Addition and subtraction are given lower precedence than multiplication and division, and all operators are left-associative. This setup is crucial for correctly parsing mathematical expressions.

```rust
lazy_static::lazy_static! {
    static ref PRATT_PARSER: PrattParser<Rule> = {
        use pest::pratt_parser::{Assoc::*,
 Op};
        use Rule::*;

        // Precedence is defined lowest to highest
        PrattParser::new()
            // Addition and subtract have equal precedence
            .op(Op::infix(add, Left) | Op::infix(subtract, Left))
            .op(Op::infix(multiply, Left) | Op::infix(divide, Left))
    };
}

```

--------------------------------

### Cargo.toml: Enabling Pest Grammar Extras Feature

Source: https://pest.rs/book/grammars/syntax

Shows how to enable the 'grammar-extras' feature for the `pest_derive` dependency in a Cargo.toml file. This feature is required for using functionalities like rule tagging.

```toml
# ...
pest_derive = { version = "2.7", features = ["grammar-extras"] }
```

--------------------------------

### Pest Rust Parser Evaluation Logic

Source: https://pest.rs/book/intro

Implements the logic to evaluate expressions parsed by the Pest grammar using Rust. It utilizes `lazy_static` for a global parser instance and defines functions to map primary and infix elements to their corresponding actions.

```rust
lazy_static! {
    static ref PRATT_PARSER: PrattParser<Rule> = {
        use Rule::*;
        use Assoc::*;

        PrattParser::new()
            .op(Op::infix(add, Left) | Op::infix(subtract, Left))
            .op(Op::infix(multiply, Left) | Op::infix(divide, Left))
            .op(Op::infix(power, Right))
    };
}

fn eval(expression: Pairs<Rule>) -> f64 {
    PRATT_PARSER
        .map_primary(|primary| match primary.as_rule() {
            Rule::num => primary.as_str().parse::<f64>().unwrap(),
            Rule::expr => eval(primary.into_inner()),
            _ => unreachable!(),
        })
        .map_infix(|lhs, op, rhs| match op.as_rule() {
            Rule::add => lhs + rhs,
            Rule::subtract => lhs - rhs,
            Rule::multiply => lhs * rhs,
            Rule::divide => lhs / rhs,
            Rule::power => lhs.powf(rhs),
            _ => unreachable!(),
        })
        .parse(expression)
}
```

--------------------------------

### Parse J Program into AST (Rust)

Source: https://pest.rs/book/examples/jlang

Parses a J program string into an Abstract Syntax Tree (AST). It uses the `pest` library to tokenize the input according to the defined grammar. Each top-level expression is parsed and wrapped in a `Print` AST node, mimicking REPL behavior. Returns a `Result` containing the AST or a parsing error.

```rust
pub fn parse(source: &str) -> Result<Vec<AstNode>, Error<Rule>> {
    let mut ast = vec![];

    let pairs = JParser::parse(Rule::program, source)?;
    for pair in pairs {
        match pair.as_rule() {
            Rule::expr => {
                ast.push(Print(Box::new(build_ast_from_expr(pair))));
            }
            _ => {} // Ignore other rules
        }
    }

    Ok(ast)
}
```

--------------------------------

### Matching and Unwrapping Parse Results

Source: https://pest.rs/book/parser_api

Demonstrates how to handle the `Result<Pairs, Error>` returned by the `Parser::parse` method. It shows using a `match` statement to process successful parses or handle errors.

```rust
// check whether parse was successful
match Parser::parse(Rule::enclosed, "(..6472..)") {
    Ok(mut pairs) => {
        let enclosed = pairs.next().unwrap();
        // ...
    }
    Err(error) => {
        // ...
    }
}
```

--------------------------------

### Pest Repetition Operators

Source: https://pest.rs/book/grammars/syntax

Demonstrates Pest's repetition operators: `*` (zero or more), `+` (one or more), and `?` (optional, zero or one). It also covers the more explicit repetition syntax using curly braces for exact counts, ranges, and minimum/maximum occurrences.

```pest
("zero" ~ "or" ~ "more")*
 ("one" | "or" | "more")+
           (^"optional")?
```

```pest
expr{n}           // exactly n repetitions
expr{m, n}        // between m and n repetitions, inclusive

expr{, n}         // at most n repetitions
expr{m, }         // at least m repetitions
```

--------------------------------

### Define Binary Operator Rules for Pest Parser

Source: https://pest.rs/book/examples/calculator

This snippet defines the grammar rules for binary operators used in expressions. It includes rules for addition, subtraction, multiplication, and division, each represented by their respective symbols.

```pest
bin_op = _{ add | subtract | multiply | divide }
    add = { "+" }
    subtract = { "-" }
    multiply = { "*" }
    divide = { "/" }
```

--------------------------------

### Define Parser with External Grammar - Rust

Source: https://pest.rs/book/grammars/grammars

This Rust code snippet defines a parser using the pest_derive crate, specifying an external grammar file located at 'parser/grammar.pest' relative to the project's 'src' directory. It assumes the pest and pest_derive crates are added as dependencies.

```Rust
use pest::Parser;
use pest_derive::Parser;

#[derive(Parser)]
#[grammar = "parser/grammar.pest"] // relative to project `src`
struct MyParser;

```

--------------------------------

### Pest Grammar: Indentation-Sensitive Parsing with Stack

Source: https://pest.rs/book/grammars/syntax

Parses indentation-sensitive languages like Python using the stack to manage indentation levels. PUSH stores the indentation whitespace, PEEK_ALL checks subsequent lines, and DROP removes indentation levels when exiting blocks.

```pest
Grammar = { SOI ~ NEWLINE* ~ BlockContent* ~ NEWLINE* ~ EOI }

NewBlock = {
    // The first line in the block
    PEEK_ALL ~ PUSH("  "+ | "\t"+) ~ BlockContent ~
    // Subsequent lines in the block
    (PEEK_ALL ~ BlockContent)* ~
    // Remove the last layer of indentation from the stack when exiting the block
    DROP
}

BlockName = { ASCII_ALPHA+ }

BlockContent = {
    BlockName ~ (NEWLINE | EOI) ~ NewBlock*
}
```

--------------------------------

### AWK Clone Performance Benchmarking Script

Source: https://pest.rs/book/examples/awk

A shell script for performance benchmarking of the AWK clone against GNU AWK. It generates a large dataset and measures the execution time of a specific filtering and aggregation task for both implementations.

```bash
# Generate test data
seq 1 100000 | awk '{print "user" $1, int(rand()*100), "dept" int(rand()*10)}' > large_dataset.txt

# Benchmark against GNU AWK
time awk '$2 > 50 {
            sum += $2
            count++
        }
        END { print sum/count }' large_dataset.txt
time ./target/release/awk-clone -p '$2 > 50 {
            sum += $2
            count++
        }
        END { print sum/count }' large_dataset.txt
```

--------------------------------

### Referencing and Recursive Pest Rules

Source: https://pest.rs/book/grammars/syntax

Shows how to reference other defined rules by their names, enabling modularity and composition. It also demonstrates the concept of recursive rules, where a rule can refer to itself, allowing for the parsing of arbitrarily nested structures.

```pest
my_rule = { "slithy " ~ other_rule }
other_rule = { "toves" }
recursive_rule = { "mimsy " ~ recursive_rule }
```

--------------------------------

### Rust Unit Tests for AWK Clone Interpreter

Source: https://pest.rs/book/examples/awk

Demonstrates unit tests for the AWK clone interpreter, specifically testing field splitting and expression evaluation logic. These tests ensure the core functionality of parsing and interpreting basic AWK constructs works as expected.

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_field_splitting() {
        let mut interpreter = Interpreter::new();
        interpreter.split_fields("Alice 25 Engineer");
        assert_eq!(interpreter.fields[0], "Alice 25 Engineer");  // $0
        assert_eq!(interpreter.fields[1], "Alice");              // $1
        assert_eq!(interpreter.fields[2], "25");                 // $2
        assert_eq!(interpreter.fields[3], "Engineer");           // $3
    }

    #[test]
    fn test_expression_evaluation() {
        let interpreter = Interpreter::new();
        // Test that "25" > "100" evaluates correctly (numerically)
        let expr = Expr::BinaryOp {
            op: BinOp::Gt,
            left: Box::new(Expr::String("25".to_string())),
            right: Box::new(Expr::String("100".to_string())),
        };
        let result = interpreter.eval_expr(&expr).unwrap();
        assert_eq!(result.to_number(), 0.0);  // 25 > 100 is false
    }
}
```

--------------------------------

### Rust Parser Initialization with Pest Derive

Source: https://pest.rs/book/examples/ini

Initializes the Pest parser in Rust using the `pest_derive` macro. This involves importing necessary traits and defining the parser struct, linking it to the grammar file.

```rust
use pest::Parser;
use pest_derive::Parser;

#[derive(Parser)]
#[grammar = "ini.pest"]
pub struct INIParser;
```

--------------------------------

### Pest Grammar: Parsing Rust Raw String Literals with Stack

Source: https://pest.rs/book/grammars/syntax

Parses Rust raw string literals by tracking the number of '#' characters before the opening quote using the stack. The PUSH("#[...]*") captures the delimiters, and POP is used to match them at the end. This prevents premature string termination.

```pest
raw_string = {
    "r" ~ PUSH("#"*) ~ "\""    // push the number signs onto the stack
    ~ raw_string_interior
    ~ "\"" ~ POP               // match a quotation mark and the number signs
}
raw_string_interior = {
    (
        !("\"" ~ PEEK)    // unless the next character is a quotation mark
                          // followed by the correct amount of number signs,
        ~ ANY             // consume one character
    )*
}
```

--------------------------------

### Basic Pest Expression Types

Source: https://pest.rs/book/grammars/syntax

Illustrates fundamental expression types used in Pest grammars for matching input. This includes literal strings, case-insensitive ASCII strings, character ranges, and matching any single Unicode character.

```pest
"a literal string"
^"ASCII case-insensitive string"
'a'..'z'
ANY
```

--------------------------------

### Global State with `lazy_static`

Source: https://pest.rs/book/examples/awk

Illustrates the use of the `lazy_static` crate to initialize and manage global static variables, ensuring they are computed only once.

```rust
// Computed once, used many times
lazy_static! {
    static ref PRATT_PARSER: PrattParser<Rule> = {
        /* ... */
    };
}
```

--------------------------------

### Main Function for JSON Parsing and Serialization

Source: https://pest.rs/book/examples/json

This `main` function orchestrates the JSON processing. It reads a file named 'data.json', parses its content using `parse_json_file`, and then serializes the resulting `JSONValue` AST back into a string using an assumed `serialize_jsonvalue` function before printing it to the console. It uses `std::fs` for file reading and `expect` for error handling.

```rust
use std::fs;

fn main() {
    let unparsed_file = fs::read_to_string("data.json").expect("cannot read file");

    let json: JSONValue = parse_json_file(&unparsed_file).expect("unsuccessful parse");

    println!("{}", serialize_jsonvalue(&json));
}
```

--------------------------------

### Extracting Rule and String Content with Pair

Source: https://pest.rs/book/parser_api

Demonstrates how to use the `Pair` type to determine the rule that produced a match and retrieve the matched string. It also shows how to access inner rules using `into_inner()`.

```rust
let pair = Parser::parse(Rule::enclosed, "(..6472..)")
    .unwrap().next().unwrap();

assert_eq!(pair.as_rule(), Rule::enclosed);
assert_eq!(pair.as_str(), "(..6472..)");

let inner_rules = pair.into_inner();
println!("{}", inner_rules); // --> [number(3, 7)]
```

--------------------------------

### Define Integer and Atom Rules for Pest Parser

Source: https://pest.rs/book/examples/calculator

This snippet defines the grammar rules for 'integer' and 'atom' in Pest. 'integer' matches one or more ASCII digits without whitespace, and 'atom' is defined as a simple integer, serving as the base unit for expressions.

```pest
integer = @{ ASCII_DIGIT+ }

atom = _{ integer }
```

--------------------------------

### AWK Interpreter Initialization and Configuration (Rust)

Source: https://pest.rs/book/examples/awk

Implements the `new` constructor for the `Interpreter` in Rust, initializing the state with default values for AWK's built-in variables (FS, OFS, RS, ORS). It also provides a `set_field_separator` method to allow external modification of the field separator, reflecting command-line options like `-F`.

```rust
impl Interpreter {
    pub fn new() -> Self {
        let mut variables = HashMap::new();
        
        // Initialize AWK's built-in variables with default values
        variables.insert("FS".to_string(), Value::String(" ".to_string()));
        variables.insert("OFS".to_string(), Value::String(" ".to_string()));
        variables.insert("RS".to_string(), Value::String("\n".to_string()));
        variables.insert("ORS".to_string(), Value::String("\n".to_string()));
        
        Self {
            variables,
            fields: Vec::new(),
            record_number: 0,
            field_separator: " ".to_string(),
            output_field_separator: " ".to_string(),
            record_separator: "\n".to_string(),
            output_record_separator: "\n".to_string(),
        }
    }
    
    // Allow external configuration of field separator (for -F command line option)
    pub fn set_field_separator(&mut self, fs: String) {
        self.field_separator = fs.clone();
        self.variables.insert("FS".to_string(), Value::String(fs));
    }
}

```

--------------------------------

### Define Parser with Inline Grammar - Rust

Source: https://pest.rs/book/grammars/grammars

This Rust code demonstrates how to define a parser using pest_derive with an inline grammar. The grammar rules are provided directly as a raw string literal to the `grammar_inline` attribute. This is useful when the grammar is small or for quick prototyping.

```Rust
use pest::Parser;
use pest_derive::Parser;

#[derive(Parser)]
#[grammar_inline = r#"// your grammar here
a = { "a" }"#]
struct MyParser;

```

--------------------------------

### Pest: Tagging Rule Components

Source: https://pest.rs/book/grammars/syntax

Shows how to attach a label or tag to a part of a Pest rule using the '#' symbol. This aids in distinguishing token types and extracting information from parse trees.

```pest
rule = { #tag = ... }
```
Tokens:
Raw