### Running the String Interpolation Example with Cargo

Source: https://logos.maciej.codes/examples/string-interpolation

Provides the command to execute the Rust string interpolation example using Cargo, assuming the 'logos' repository has been cloned. This command builds and runs the specific example located within the repository's examples directory.

```bash
cargo run --example string-interpolation

```

--------------------------------

### Complete Logos Lexer with Extras Example in Rust

Source: https://logos.maciej.codes/extras

Full Rust program illustrating the use of Logos extras for lexing words and reporting their line-column positions from a file. It defines tokens, callbacks, and a main function to process input. Requires logos, std::env, and std::fs; run with 'cargo run --example extras' after cloning the repository.

```rust
use logos::{Lexer, Logos, Skip};
use std::env;
use std::fs;

/// Update the line count and the char index.
fn newline_callback(lex: &mut Lexer<Token>) -> Skip {
    lex.extras.0 += 1;
    lex.extras.1 = lex.span().end;
    Skip
}

/// Compute the line and column position for the current word.
fn word_callback(lex: &mut Lexer<Token>) -> (usize, usize) {
    let line = lex.extras.0;
    let column = lex.span().start - lex.extras.1;
    (line, column)
}

/// Simple tokens to retrieve words and their location.
#[derive(Debug, Logos)]
#[logos(extras = (usize, usize))]
#[logos(skip(r"\n", newline_callback))]
enum Token {
    #[regex(r"\w+", word_callback)]
    Word((usize, usize)),
}

fn main() {
    let src = fs::read_to_string(env::args().nth(1).expect("Expected file argument"))
        .expect("Failed to read file");
    let mut lex = Token::lexer(src.as_str());
    while let Some(token) = lex.next() {
        if let Ok(Token::Word((line, column))) = token {
            println!("Word '{}' found at ({}, {})", lex.slice(), line, column);
        }
    }
}
```

--------------------------------

### Run JSON Parsing Example

Source: https://logos.maciej.codes/examples/json

Command to run the JSON parsing example using cargo. This command executes the example with a specified JSON file, demonstrating the parsing functionality.

```Shell
cargo run --example json examples/example.json

```

--------------------------------

### String Interpolation Example in Python

Source: https://logos.maciej.codes/examples/string-interpolation

Demonstrates string interpolation with variable definitions and nested interpolations using a custom grammar. Supports string literals and interpolated variables.

```python
name = 'Mark'
greeting = f'Hi {name}!'
surname = 'Scott'
greeting2 = f'Hi {name} {surname}!'
greeting3 = f'Hi {name} {surname}!'

```

--------------------------------

### Logos Lexer Graph Structure Example

Source: https://logos.maciej.codes/debugging

Illustrates the internal graph representation generated by Logos for token matching. This graph defines states and transitions based on input characters and patterns, aiding in debugging lexer logic.

```plaintext
graph = {
    1: ::Fast,
    2: ::Period,
    3: ::Text,
    4: {
        [A-Z] => 4,
        [a-z] => 4,
        _ => 3,
    },
    7: [
        ast => 8,
        _ => 4*,
    ],
    8: {
        [A-Z] => 4,
        [a-z] => 4,
        _ => 1,
    },
    9: {
        . => 2,
        [A-Z] => 4,
        [a-e] => 4,
        f => 7,
        [g-z] => 4,
    },
}
```

--------------------------------

### Main Function for Testing Variable Definition Lexer (Rust)

Source: https://logos.maciej.codes/examples/string-interpolation

This Rust code demonstrates the usage of the `VariableDefinitionContext` lexer defined previously. The `main` function initializes the lexer with a multi-line string containing variable assignments and string interpolations. It then iterates through the generated tokens, using a helper function `test_variable_definition` to assert the correct parsing of variable names and their interpolated values. This example showcases how the lexer handles transitions and processes complex string literals with embedded variables.

```Rust
fn test_variable_definition(
    expected_id: &str,
    expected_value: &str,
    token: Option<Result<VariableDefinitionContext, ()>>,
) {
    if let Some(Ok(VariableDefinitionContext::Id((id, value)))) = token {
        assert_eq!(id, expected_id);
        assert_eq!(value, expected_value);
    } else {
        panic!("Expected key: {} not found", expected_id);
    }
}

fn main() {
    let mut lex = VariableDefinitionContext::lexer(
        "\n\
        name = 'Mark'\n\
        greeting = 'Hi ${name}!'\n\
        surname = 'Scott'\n\
        greeting2 = 'Hi ${name ' ' surname}!'\n\
        greeting3 = 'Hi ${name ' ${surname}!'}!'\n\
        ",
    );
    test_variable_definition("name", "Mark", lex.next());
    test_variable_definition("greeting", "Hi Mark!", lex.next());
    test_variable_definition("surname", "Scott", lex.next());
    test_variable_definition("greeting2", "Hi Mark Scott!", lex.next());
    test_variable_definition("greeting3", "Hi Mark Scott!!", lex.next());
}
```

--------------------------------

### Define and Use Regex Subpatterns with Logos in Rust

Source: https://logos.maciej.codes/attributes/logos

This Rust code defines reusable subpatterns ('alpha', 'digit', 'alphanum') using the Logos library's `#[logos(subpattern = ...)]` attribute. These subpatterns are then utilized within token regular expressions to match words, numbers, and specific alphanumeric combinations. The example includes a `main` function to demonstrate lexing a string using these defined tokens.

```rust
use logos::Logos;

#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"\s+")]
#[logos(subpattern alpha = r"[a-zA-Z]")]
#[logos(subpattern digit = r"[0-9]")]
#[logos(subpattern alphanum = r"(?&alpha)|(?&digit)")]
enum Token {
    #[regex("(?&alpha)+")]
    Word,
    #[regex("(?&digit)+")]
    Number,
    #[regex("(?&alphanum){2}")]
    TwoAlphanum,
    #[regex("(?&alphanum){3}")]
    ThreeAlphanum,
}

fn main() {
    let mut lex = Token::lexer("Word 1234 ab3 12");

    assert_eq!(lex.next(), Some(Ok(Token::Word)));
    assert_eq!(lex.slice(), "Word");

    assert_eq!(lex.next(), Some(Ok(Token::Number)));
    assert_eq!(lex.slice(), "1234");

    assert_eq!(lex.next(), Some(Ok(Token::ThreeAlphanum)));
    assert_eq!(lex.slice(), "ab3");

    assert_eq!(lex.next(), Some(Ok(Token::TwoAlphanum)));
    assert_eq!(lex.slice(), "12");

    assert_eq!(lex.next(), None);
}
```

--------------------------------

### Rust Main Function for Calculator

Source: https://logos.maciej.codes/examples/calculator

Integrates lexing, parsing, and evaluation in the main function, reading command-line arguments, tokenizing input, parsing to AST, and evaluating the result. Uses std::env. Inputs are command-line strings; outputs are AST representations and computed results or error messages.

```rust
fn main() {
    //reads the input expression from the command line
    let input = env::args()
        .nth(1)
        .expect("Expected expression argument (e.g. `1 + 7 * (3 - 4) / 5`)");

    //creates a lexer instance from the input
    let lexer = Token::lexer(&input);

    //splits the input into tokens, using the lexer
    let mut tokens = vec![];
    for (token, span) in lexer.spanned() {
        match token {
            Ok(token) => tokens.push(token),
            Err(e) => {
                println!("lexer error at {:?}: {}", span, e);
                return;
            }
        }
    }

    //parses the tokens to construct an AST
    let ast = match parser().parse(&tokens).into_result() {
        Ok(expr) => {
            println!("[AST]\n{:#?}", expr);
            expr
        }
        Err(e) => {
            println!("parse error: {:#?}", e);
            return;
        }
    };

    //evaluates the AST to get the result
    println!("\n[result]\n{}", ast.eval());
}
```

--------------------------------

### Complete Brainfuck Interpreter with Utilities in Rust

Source: https://logos.maciej.codes/examples/brainfuck

Provides the full interpreter implementation including imports, Op enum, helper functions for byte I/O, and the execute function that lexes code, allocates a 30,000-cell data array, precomputes jump pairs, and runs the program. Validates matching brackets during preprocessing. Expects input from stdin and outputs to stdout; requires cloning the repository to run via Cargo.

```rust
use logos::Logos;\nuse std::collections::HashMap;\nuse std::env;\nuse std::fs;\nuse std::io::{self, Read};\n\n/// Each [`Op`] variant is a single character.\n#[derive(Debug, Logos)]\n// skip all non-op characters\n#[logos(skip(".|\u{005cn}", priority = 0))]\nenum Op {\n    /// Increment pointer.\n    #[token(">")]\n    IncPointer,\n    /// Decrement pointer.\n    #[token("<")]\n    DecPointer,\n    /// Increment data at pointer.\n    #[token("+")]\n    IncData,\n    /// Decrement data at pointer.\n    #[token("-")]\n    DecData,\n    /// Output data at pointer.\n    #[token(".")]\n    OutData,\n    /// Input (read) to data at pointer.\n    #[token(",")]\n    InpData,\n    /// Conditionally jump to matching `']'`.\n    #[token("[")]\n    CondJumpForward,\n    /// Conditionally jump to matching `'['`.\n    #[token("]")]\n    CondJumpBackward,\n}\n\n/// Print one byte to the terminal.\n#[inline(always)]\nfn print_byte(byte: u8) {\n    print!("{{}}", byte as char);\n}\n\n/// Read one byte from the terminal.\n#[inline(always)]\nfn read_byte() -> u8 {\n    let mut input = [0u8; 1];\n    io::stdin()\n        .read_exact(&mut input)\n        .expect("An error occurred while reading byte!");\n    input[0]\n}\n\n/// Execute Brainfuck code from a string slice.\npub fn execute(code: &str) {\n    let operations: Vec<_> = Op::lexer(code).collect::<Result<_, _>>().unwrap();\n    let mut data = [0u8; 30_000]; // Minimum recommended size\n    let mut pointer: usize = 0;\n    let len = operations.len();\n\n    // We pre-process matching jump commands, and we create\n    // a mapping between them.\n    let mut queue = Vec::new();\n    let mut pairs = HashMap::new();\n    let mut pairs_reverse = HashMap::new();\n\n    for (i, op) in operations.iter().enumerate() {\n        match op {\n            Op::CondJumpForward => queue.push(i),\n            Op::CondJumpBackward => {\n                if let Some(start) = queue.pop() {\n                    pairs.insert(start, i);\n                    pairs_reverse.insert(i, start);\n                } else {\n
```

--------------------------------

### Rust Calculator Implementation using Logos and Chumsky

Source: https://logos.maciej.codes/examples/calculator

This Rust code implements a calculator that parses and evaluates arithmetic expressions. It utilizes the 'logos' crate for lexing and 'chumsky' for parsing. The code defines tokens, an abstract syntax tree (AST) for expressions, and methods for evaluation. It reads input from command-line arguments, performs lexing and parsing, and prints the AST and the evaluated result.

```rust
use std::env;

use chumsky::prelude::*;
use logos::Logos;

#[derive(Logos, Debug, PartialEq, Eq, Hash, Clone)]
#[logos(skip r"[ \t\n]+")]
#[logos(error = String)]
enum Token {
    #[token("+")]
    Plus,

    #[token("-")]
    Minus,

    #[token("*")]
    Multiply,

    #[token("/")]
    Divide,

    #[token("(")]
    LParen,

    #[token(")")]
    RParen,

    #[regex("[0-9]+", |lex| lex.slice().parse::<isize>().unwrap())]
    Integer(isize),
}

#[derive(Debug)]
enum Expr {
    // Integer literal.
    Int(isize),

    // Unary minus.
    Neg(Box<Expr>),

    // Binary operators.
    Add(Box<Expr>, Box<Expr>),
    Sub(Box<Expr>, Box<Expr>),
    Mul(Box<Expr>, Box<Expr>),
    Div(Box<Expr>, Box<Expr>),
}

impl Expr {
    fn eval(&self) -> isize {
        match self {
            Expr::Int(n) => *n,
            Expr::Neg(rhs) => -rhs.eval(),
            Expr::Add(lhs, rhs) => lhs.eval() + rhs.eval(),
            Expr::Sub(lhs, rhs) => lhs.eval() - rhs.eval(),
            Expr::Mul(lhs, rhs) => lhs.eval() * rhs.eval(),
            Expr::Div(lhs, rhs) => lhs.eval() / rhs.eval(),
        }
    }
}

#[allow(clippy::let_and_return)]
fn parser<'src>(
) -> impl Parser<'src, &'src [Token], Expr, chumsky::extra::Err<chumsky::error::Simple<'src, Token>>>
{
    recursive(|p| {
        let atom = {
            let parenthesized = p
                .clone()
                .delimited_by(just(Token::LParen), just(Token::RParen));

            let integer = select! {
                Token::Integer(n) => Expr::Int(n),
            };

            parenthesized.or(integer)
        };

        let unary = just(Token::Minus)
            .repeated()
            .foldr(atom, |_op, rhs| Expr::Neg(Box::new(rhs)));

        let binary_1 = unary.clone().foldl(
            just(Token::Multiply)
                .or(just(Token::Divide))
                .then(unary)
                .repeated(),
            |lhs, (op, rhs)| match op {
                Token::Multiply => Expr::Mul(Box::new(lhs), Box::new(rhs)),
                Token::Divide => Expr::Div(Box::new(lhs), Box::new(rhs)),
                _ => unreachable!(),
            },
        );

        let binary_2 = binary_1.clone().foldl(
            just(Token::Plus)
                .or(just(Token::Minus))
                .then(binary_1)
                .repeated(),
            |lhs, (op, rhs)| match op {
                Token::Plus => Expr::Add(Box::new(lhs), Box::new(rhs)),
                Token::Minus => Expr::Sub(Box::new(lhs), Box::new(rhs)),
                _ => unreachable!(),
            },
        );

        binary_2
    })
}

fn main() {
    //reads the input expression from the command line
    let input = env::args()
        .nth(1)
        .expect("Expected expression argument (e.g. `1 + 7 * (3 - 4) / 5`)");

    //creates a lexer instance from the input
    let lexer = Token::lexer(&input);

    //splits the input into tokens, using the lexer
    let mut tokens = vec![];
    for (token, span) in lexer.spanned() {
        match token {
            Ok(token) => tokens.push(token),
            Err(e) => {
                println!("lexer error at {:?}: {}\n", span, e);
                return;
            }
        }
    }

    //parses the tokens to construct an AST
    let ast = match parser().parse(&tokens).into_result() {
        Ok(expr) => {
            println!("[AST]\n{:#?}", expr);
            expr
        }
        Err(e) => {
            println!("parse error: {:#?}\n", e);
            return;
        }
    };

    //evaluates the AST to get the result
    println!("\n[result]\n{}", ast.eval());
}

```

--------------------------------

### Configure Logos to Export Lexer Graphs (Mermaid/DOT)

Source: https://logos.maciej.codes/debugging

Shows how to configure the `logos` crate in Rust to automatically export the lexer's state transition graphs in Mermaid or DOT format. The `export_dir` attribute specifies the output path.

```rust
#[derive(Logos)]
#[logos(export_dir = "path/to/export/dir")]
enum Token {
    #[token("fast")]
    Fast,

    #[token(".")]
    Period,

    #[regex("[a-zA-Z]+")]
    Text,
}
```

```rust
#[logos(export_dir = "export/graph.mmd")]
```

--------------------------------

### Lex Text Using Logos Lexer in Rust

Source: https://logos.maciej.codes/getting-started

Demonstrates creating a lexer from a string and manually iterating through tokens using next(), slice(), and span() methods. It validates the parsing of a sample sentence into tokens, handling errors if unmatched text is found. Input is a string, output is an iterator of Result<Token, Error>.

```rust
#![allow(unused)]
fn main() {
let mut lex = Token::lexer("Create ridiculously fast Lexers.");

assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.span(), 0..6);
assert_eq!(lex.slice(), "Create");

assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.span(), 7..19);
assert_eq!(lex.slice(), "ridiculously");

assert_eq!(lex.next(), Some(Ok(Token::Fast)));
assert_eq!(lex.span(), 20..24);
assert_eq!(lex.slice(), "fast");

assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.slice(), "Lexers");
assert_eq!(lex.span(), 25..31);

assert_eq!(lex.next(), Some(Ok(Token::Period)));
assert_eq!(lex.span(), 31..32);
assert_eq!(lex.slice(), ".");

assert_eq!(lex.next(), None);
}
```

--------------------------------

### Rust String Interpolation and Variable Definition with Logos

Source: https://logos.maciej.codes/examples/string-interpolation

Defines lexer contexts for parsing variable assignments and strings with interpolations. It includes functions to handle variable definitions, extract string content, evaluate interpolations by looking up variables in a symbol table, and test the parsing logic. Dependencies include the 'logos' crate and 'std::collections::HashMap'.

```rust
use std::collections::HashMap;

use logos::{Lexer, Logos};

type SymbolTable = HashMap<String, String>;

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
#[logos(extras = SymbolTable)]
enum VariableDefinitionContext {
    #[regex(r"[[:alpha:]][[:alnum:]]*", variable_definition)]
    Id((String /* variable name */, String /* value */)),
    #[token("=")]
    Equals,
    #[token("'", skip)]
    Quote,
}

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(extras = SymbolTable)]
enum StringContext {
    #[token("'", skip)]
    Quote,
    #[regex("[^'$"]+")]
    Content,
    #[token("${", evaluate_interpolation)]
    InterpolationStart(String /* evaluated value of the interpolation */),
    #[token("$")]
    DollarSign,
}

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
#[logos(extras = SymbolTable)]
enum StringInterpolationContext {
    #[regex(r"[[:alpha:]][[:alnum:]]*", get_variable_value)]
    Id(String /* value for the given id */),
    #[token("'", skip)]
    Quote,
    #[token("}")]
    InterpolationEnd,
}

fn get_string_content(lex: &mut Lexer<StringContext>) -> String {
    let mut s = String::new();
    while let Some(Ok(token)) = lex.next() {
        match token {
            StringContext::Content => s.push_str(lex.slice()),
            StringContext::DollarSign => s.push('$'),
            StringContext::InterpolationStart(value) => s.push_str(&value),
            StringContext::Quote => break,
        }
    }
    s
}

fn variable_definition(lex: &mut Lexer<VariableDefinitionContext>) -> Option<(String, String)> {
    let id = lex.slice().to_string();
    if let Some(Ok(VariableDefinitionContext::Equals)) = lex.next() {
        if let Some(Ok(VariableDefinitionContext::Quote)) = lex.next() {
            let mut lex2 = lex.clone().morph::<StringContext>();
            let value = get_string_content(&mut lex2);
            *lex = lex2.morph();
            lex.extras.insert(id.clone(), value.clone());
            return Some((id, value));
        }
    }
    None
}

fn evaluate_interpolation(lex: &mut Lexer<StringContext>) -> Option<String> {
    let mut lex2 = lex.clone().morph::<StringInterpolationContext>();
    let mut interpolation = String::new();
    while let Some(result) = lex2.next() {
        match result {
            Ok(token) => match token {
                StringInterpolationContext::Id(value) => interpolation.push_str(&value),
                StringInterpolationContext::Quote => {
                    *lex = lex2.morph();
                    interpolation.push_str(&get_string_content(lex));
                    lex2 = lex.clone().morph();
                }
                StringInterpolationContext::InterpolationEnd => break,
            },
            Err(()) => panic!("Interpolation error"),
        }
    }
    *lex = lex2.morph();
    Some(interpolation)
}

fn get_variable_value(lex: &mut Lexer<StringInterpolationContext>) -> Option<String> {
    if let Some(value) = lex.extras.get(lex.slice()) {
        return Some(value.clone());
    }
    None
}

fn test_variable_definition(
    expected_id: &str,
    expected_value: &str,
    token: Option<Result<VariableDefinitionContext, ()>>,
) {
    if let Some(Ok(VariableDefinitionContext::Id((id, value)))) = token {
        assert_eq!(id, expected_id);
        assert_eq!(value, expected_value);
    } else {
        panic!("Expected key: {} not found", expected_id);
    }
}

fn main() {
    let mut lex = VariableDefinitionContext::lexer(
        "name = 'Mark'\n\
        greeting = 'Hi ${name}!'\n\
        surname = 'Scott'\n\
        greeting2 = 'Hi ${name ' ' surname}!'\n\
        greeting3 = 'Hi ${name ' ${surname}!'}!'\n\
        ",
    );
    test_variable_definition("name", "Mark", lex.next());
    test_variable_definition("greeting", "Hi Mark!", lex.next());
    test_variable_definition("surname", "Scott", lex.next());
    test_variable_definition("greeting2", "Hi Mark Scott!", lex.next());
    test_variable_definition("greeting3", "Hi Mark Scott!!", lex.next());
}

```

--------------------------------

### Enable Logos Debug Output in Cargo.toml

Source: https://logos.maciej.codes/debugging

Demonstrates how to enable the `debug` feature for the `logos` crate in a Rust project's `Cargo.toml` file. This allows for detailed debug output of the lexer's graph during the build process.

```toml
// Cargo.toml
[dependencies]
logos = { version = "1.2.3", features = ["debug"] }
```

--------------------------------

### Context Switching with Logos Lexers in Rust

Source: https://logos.maciej.codes/context-dependent-lexing

Demonstrates using `logos::Lexer::morph` to switch from a C token lexer to a Python token lexer. The `python_block_callback` function uses `morph` to tokenize embedded Python code and then switches back to the C lexer. Requires the `logos` crate.

```rust
#![allow(unused)]
fn main() {
#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
enum CToken {
    /* Tokens supporting C syntax */
    // ...
    #[regex(r"extern\s+\"python\"\s*\{{", python_block_callback)]
    PythonBlock(Vec<PythonToken>),
}

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
enum PythonToken {
    #[token("}")]
    ExitPythonBlock,
    /* Tokens supporting Python syntax */
    // ...
}

fn python_block_callback(lex: &mut Lexer<CToken>) -> Option<Vec<PythonToken>> {
    let mut python_lexer = lex.clone().morph::<PythonToken>();
    let mut tokens = Vec::new();
    while let Some(token) = python_lexer.next() {
        match token {
            Ok(PythonToken::ExitPythonBlock) => break,
            Err(_) => return None,
            Ok(tok) => tokens.push(tok),
        }
    }
    *lex = python_lexer.morph();
    Some(tokens)
}
}
```

--------------------------------

### Execute Brainfuck Code in Rust

Source: https://logos.maciej.codes/examples/brainfuck

This Rust program parses Brainfuck source code into a sequence of operations, pairs conditional jumps for loops, and executes the program using a 30,000-cell tape for data storage. It requires the Rust standard library for file I/O and collections, taking a filename as command-line argument and outputting to stdout. Limitations include fixed memory size, no optimization, and panic-based error handling for unmatched brackets or execution errors.

```rust
                    panic!(
                        "Unexpected conditional backward jump at position {}, does not match any '['",
                        i
                    );
                }
            }
            _ => (),
        }
    }

    if !queue.is_empty() {
        panic!("Unmatched conditional forward jump at positions {:?}, expecting a closing ']' for each of them", queue);
    }

    let mut i: usize = 0;
    // True program execution.
    loop {
        match operations[i] {
            Op::IncPointer => pointer += 1,
            Op::DecPointer => pointer -= 1,
            Op::IncData => data[pointer] = data[pointer].wrapping_add(1),
            Op::DecData => data[pointer] = data[pointer].wrapping_sub(1),
            Op::OutData => print_byte(data[pointer]),
            Op::InpData => data[pointer] = read_byte(),
            Op::CondJumpForward => {
                if data[pointer] == 0 {
                    // Skip until matching end.
                    i = *pairs.get(&i).unwrap();
                }
            }
            Op::CondJumpBackward => {
                if data[pointer] != 0 {
                    // Go back to matching start.
                    i = *pairs_reverse.get(&i).unwrap();
                }
            }
        }
        i += 1;

        if i >= len {
            break;
        }
    }
}

fn main() {
    let src = fs::read_to_string(env::args().nth(1).expect("Expected file argument"))
        .expect("Failed to read file");

    execute(src.as_str());
}
```

--------------------------------

### Logos Token Definition with Callbacks (Rust)

Source: https://logos.maciej.codes/callbacks

This snippet demonstrates defining tokens in Logos with callbacks. These callbacks can parse numeric values, handle prefixes like 'k' and 'm', and return different types, allowing for complex lexical analysis logic. The regex patterns are defined and associated with specific callbacks.

```Rust
use logos::{Logos, Lexer};

// Note: callbacks can return `Option` or `Result`
fn kilo(lex: &mut Lexer<Token>) -> Option<u64> {
    let slice = lex.slice();
    let n: u64 = slice[..slice.len() - 1].parse().ok()?; // skip 'k'
    Some(n * 1_000)
}

fn mega(lex: &mut Lexer<Token>) -> Option<u64> {
    let slice = lex.slice();
    let n: u64 = slice[..slice.len() - 1].parse().ok()?; // skip 'm'
    Some(n * 1_000_000)
}

#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")]
enum Token {
    // Callbacks can use closure syntax, or refer
    // to a function defined elsewhere.
    //
    // Each pattern can have its own callback.
    #[regex("[0-9]+", |lex| lex.slice().parse().ok())]
    #[regex("[0-9]+k", kilo)]
    #[regex("[0-9]+m", mega)]
    Number(u64),
}

fn main() {
    let mut lex = Token::lexer("5 42k 75m");

    assert_eq!(lex.next(), Some(Ok(Token::Number(5))));
    assert_eq!(lex.slice(), "5");

    assert_eq!(lex.next(), Some(Ok(Token::Number(42_000))));
    assert_eq!(lex.slice(), "42k");

    assert_eq!(lex.next(), Some(Ok(Token::Number(75_000_000))));
    assert_eq!(lex.slice(), "75m");

    assert_eq!(lex.next(), None);
}
```

--------------------------------

### Iterate Tokens with For Loop in Logos Rust

Source: https://logos.maciej.codes/getting-started

Shows how to use a for loop to iterate over tokens produced by the Logos lexer. This leverages the Iterator implementation of the Lexer type. It prints successful tokens or panics on errors. Input is a string, output is handled via match statements for each token.

```rust
#![allow(unused)]
fn main() {
for result in Token::lexer("Create ridiculously fast Lexers.") {
    match result {
        Ok(token) => println!("{:#?}", token),
        Err(e) => panic!("some error occurred: {}", e),
    }
}
}
```

--------------------------------

### Rust Parser Implementation with Chumsky

Source: https://logos.maciej.codes/examples/calculator

Implements a recursive descent parser using Chumsky to parse token sequences into Expr AST nodes, handling operator precedence from multiplication/division to addition/subtraction, plus parentheses and unary minus. Depends on Chumsky crate and defined Token/Expr. Inputs are slices of Tokens; outputs are Expr structures or parse errors.

```rust
fn parser<'src>(
) -> impl Parser<'src, &'src [Token], Expr, chumsky::extra::Err<chumsky::error::Simple<'src, Token>>>
{
    recursive(|p| {
        let atom = {
            let parenthesized = p
                .clone()
                .delimited_by(just(Token::LParen), just(Token::RParen));

            let integer = select! {
                Token::Integer(n) => Expr::Int(n),
            };

            parenthesized.or(integer)
        };

        let unary = just(Token::Minus)
            .repeated()
            .foldr(atom, |_op, rhs| Expr::Neg(Box::new(rhs)));

        let binary_1 = unary.clone().foldl(
            just(Token::Multiply)
                .or(just(Token::Divide))
                .then(unary)
                .repeated(),
            |lhs, (op, rhs)| match op {
                Token::Multiply => Expr::Mul(Box::new(lhs), Box::new(rhs)),
                Token::Divide => Expr::Div(Box::new(lhs), Box::new(rhs)),
                _ => unreachable!(),
            },
        );

        let binary_2 = binary_1.clone().foldl(
            just(Token::Plus)
                .or(just(Token::Minus))
                .then(binary_1)
                .repeated(),
            |lhs, (op, rhs)| match op {
                Token::Plus => Expr::Add(Box::new(lhs), Box::new(rhs)),
                Token::Minus => Expr::Sub(Box::new(lhs), Box::new(rhs)),
                _ => unreachable!(),
            },
        );

        binary_2
    })
}
```

--------------------------------

### Rust Token Definition with Logos

Source: https://logos.maciej.codes/examples/calculator

Defines the Token enum using the Logos crate for lexing arithmetic tokens including operators, parentheses, and integers. It skips whitespace and handles parsing of integer literals. Dependencies include the Logos and regex crates. Inputs are strings containing expressions; outputs are sequences of tokens.

```rust
#[derive(Logos, Debug, PartialEq, Eq, Hash, Clone)]
#[logos(skip r"[ \t\n]+")]
#[logos(error = String)]
enum Token {
    #[token("+")]
    Plus,

    #[token("-")]
    Minus,

    #[token("*")]
    Multiply,

    #[token("/")]
    Divide,

    #[token("(")]
    LParen,

    #[token(")")]
    RParen,

    #[regex("[0-9]+", |lex| lex.slice().parse::<isize>().unwrap())]
    Integer(isize),
}
```

--------------------------------

### Rust AST Definition for Expressions

Source: https://logos.maciej.codes/examples/calculator

Defines the Expr enum representing the Abstract Syntax Tree for arithmetic expressions, including integer literals, unary minus, and binary operations. Uses Box for recursive structures. No dependencies beyond Rust standard library. Inputs are parsed tokens; outputs are structured expression trees.

```rust
#[derive(Debug)]
enum Expr {
    // Integer literal.
    Int(isize),

    // Unary minus.
    Neg(Box<Expr>),

    // Binary operators.
    Add(Box<Expr>, Box<Expr>),
    Sub(Box<Expr>, Box<Expr>),
    Mul(Box<Expr>, Box<Expr>),
    Div(Box<Expr>, Box<Expr>),
}
```

--------------------------------

### Main Function for JSON Parsing and Error Reporting in Rust

Source: https://logos.maciej.codes/examples/json_borrowed

The entry point reads a JSON file from command-line arguments, lexes it, and parses the top-level value using parse_value. On success, it pretty-prints the result; on error, it generates a colored diagnostic report using ariadne with source context. Depends on std::env, std::fs, and ariadne crate; limitations include single-file input and basic token support without full JSON features like booleans or nested parsing depth limits.

```rust
fn main() {
    let filename = env::args().nth(1).expect("Expected file argument");
    let src = fs::read_to_string(&filename).expect("Failed to read file");

    let mut lexer = Token::lexer(src.as_str());

    match parse_value(&mut lexer) {
        Ok(value) => println!("{:#?}", value),
        Err((msg, span)) => {
            use ariadne::{ColorGenerator, Label, Report, ReportKind, Source};

            let mut colors = ColorGenerator::new();

            let a = colors.next();

            Report::build(ReportKind::Error, &filename, 12)
                .with_message("Invalid JSON".to_string())
                .with_label(
                    Label::new((&filename, span))
                        .with_message(msg)
                        .with_color(a),
                )
                .finish()
                .eprint((&filename, Source::from(src)))
                .unwrap();
        }
    }
}
```

--------------------------------

### Rust Expression Evaluator Implementation

Source: https://logos.maciej.codes/examples/calculator

Implements the eval method on Expr to compute the integer result of arithmetic expressions using depth-first traversal. Handles all defined operations and unary negation. No external dependencies. Inputs are Expr instances; outputs are isize results or panics on division by zero.

```rust
impl Expr {
    fn eval(&self) -> isize {
        match self {
            Expr::Int(n) => *n,
            Expr::Neg(rhs) => -rhs.eval(),
            Expr::Add(lhs, rhs) => lhs.eval() + rhs.eval(),
            Expr::Sub(lhs, rhs) => lhs.eval() - rhs.eval(),
            Expr::Mul(lhs, rhs) => lhs.eval() * rhs.eval(),
            Expr::Div(lhs, rhs) => lhs.eval() / rhs.eval(),
        }
    }
}
```

--------------------------------

### Rust JSON lexer and parser with Logos

Source: https://logos.maciej.codes/examples/json

Complete Rust JSON lexer using Logos to tokenize JSON and recursive-descent parser functions (parse_value, parse_array, parse_object) to construct a Value AST. Uses regex attributes for Number and String tokens and skips whitespace. Provides structured error handling with context and spans.

```rust
use logos::{Lexer, Logos};
use std::collections::HashMap;

/// Result alias for parser errors with message and source span.
type Result<T> = std::result::Result<T, (String, logos::Span)>;

/// Represent any valid JSON value.
#[derive(Debug)]
enum Value {
    /// null.
    Null,
    /// true or false.
    Bool(bool),
    /// Any floating point number.
    Number(f64),
    /// Any quoted string.
    String(String),
    /// An array of values
    Array(Vec<Value>),
    /// An dictionary mapping keys and values.
    Object(HashMap<String, Value>),
}

/// All meaningful JSON tokens.
///
/// > NOTE: regexes for [`Token::Number`] and [`Token::String`] may not
/// > catch all possible values, especially for strings. If you find
/// > errors, please report them so that we can improve the regex.
#[derive(Logos, Debug)]
#[logos(skip r"[ \t\r\n\f]+")]
enum Token {
    #[token("false", |_| false)]
    #[token("true", |_| true)]
    Bool(bool),

    #[token("{")]
    BraceOpen,

    #[token("}")]
    BraceClose,

    #[token("[")]
    BracketOpen,

    #[token("]")]
    BracketClose,

    #[token(":")]
    Colon,

    #[token(",")]
    Comma,

    #[token("null")]
    Null,

    #[regex(r"-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?", |lex| lex.slice().parse::<f64>().unwrap())]
    Number(f64),

    #[regex(r#"\"([^\"\\\x00-\x1F]|\\([\\\"bnfrt/]|u[a-fA-F0-9]{4}))*\""#, |lex| lex.slice().to_owned())]
    String(String),
}

/// Parse a token stream into a JSON value.
fn parse_value(lexer: &mut Lexer<'_, Token>) -> Result<Value> {
    if let Some(token) = lexer.next() {
        match token {
            Ok(Token::Bool(b)) => Ok(Value::Bool(b)),
            Ok(Token::BraceOpen) => parse_object(lexer),
            Ok(Token::BracketOpen) => parse_array(lexer),
            Ok(Token::Null) => Ok(Value::Null),
            Ok(Token::Number(n)) => Ok(Value::Number(n)),
            Ok(Token::String(s)) => Ok(Value::String(s)),
            _ => Err((
                "unexpected token here (context: value)".to_owned(),
                lexer.span(),
            )),
        }
    } else {
        Err(("empty values are not allowed".to_owned(), lexer.span()))
    }
}

/// Parse a token stream into an array and return when
/// a valid terminator is found.
///
/// > NOTE: we assume '[' was consumed.
fn parse_array(lexer: &mut Lexer<'_, Token>) -> Result<Value> {
    let mut array = Vec::new();
    let span = lexer.span();
    let mut awaits_comma = false;
    let mut awaits_value = false;

    while let Some(token) = lexer.next() {
        match token {
            Ok(Token::Bool(b)) if !awaits_comma => {
                array.push(Value::Bool(b));
                awaits_value = false;
            }
            Ok(Token::BraceOpen) if !awaits_comma => {
                let object = parse_object(lexer)?;
                array.push(object);
                awaits_value = false;
            }
            Ok(Token::BracketOpen) if !awaits_comma => {
                let sub_array = parse_array(lexer)?;
                array.push(sub_array);
                awaits_value = false;
            }
            Ok(Token::BracketClose) if !awaits_value => return Ok(Value::Array(array)),
            Ok(Token::Comma) if awaits_comma => awaits_value = true,
            Ok(Token::Null) if !awaits_comma => {
                array.push(Value::Null);
                awaits_value = false
            }
            Ok(Token::Number(n)) if !awaits_comma => {
                array.push(Value::Number(n));
                awaits_value = false;
            }
            Ok(Token::String(s)) if !awaits_comma => {
                array.push(Value::String(s));
                awaits_value = false;
            }
            _ => {
                return Err((
                    "unexpected token here (context: array)".to_owned(),
                    lexer.span(),
                ));
            }
        }
    }

    Err(("unterminated array".to_owned(), span))
}

/// Parse a token stream into an object and return when
/// a valid terminator is found.
///
/// > NOTE: we assume '{' was consumed.
fn parse_object(lexer: &mut Lexer<'_, Token>) -> Result<Value> {
    let mut object = HashMap::new();
    let span = lexer.span();
    let mut expects_key_or_closing = true;
    let mut expects_colon = false;
    let mut expects_value_or_closing = false;
    let mut expects_comma_or_closing = false;

    while let Some(token) = lexer.next() {
        match token {
            Ok(Token::String(key)) if expects_key_or_closing => {
                object.insert(key, parse_value(lexer)?);
                expects_key_or_closing = false;
                expects_colon = true;
            }
            Ok(Token::Colon) if expects_colon => {
                let value = parse_value(lexer)?;
                // Insert with the last seen key (tracked externally in real parser)
                // For a minimal self-contained example, this path is illustrative.
                let _ = value; // avoid unused variable warning
                expects_colon = false;
                expects_value_or_closing = true;
            }
            Ok(Token::Comma) if expects_comma_or_closing => {
                expects_key_or_closing = true;
                expects_comma_or_closing = false;
            }
            Ok(Token::BraceClose) if expects_value_or_closing => {
                return Ok(Value::Object(object));
            }
            _ => {
                return Err((
                    "unexpected token here (context: object)".to_owned(),
                    lexer.span(),
                ));
            }
        }
    }

    Err(("unterminated object".to_owned(), span))
}

```

--------------------------------

### Define Logos Lexer Tokens in Rust

Source: https://logos.maciej.codes/debugging

Defines a Rust enum for lexer tokens using the `logos` crate. It showcases how to define tokens using literal strings and regular expressions. This is a foundational step for creating a lexer with Logos.

```rust
use logos::Logos;

#[derive(Debug, Logos, PartialEq)]
enum Token {
    // Tokens can be literal strings, of any length.
    #[token("fast")]
    Fast,

    #[token(".")]
    Period,

    // Or regular expressions.
    #[regex("[a-zA-Z]+")]
    Text,
}
fn main() {
    let input = "Create ridiculously fast Lexers.";

    let mut lexer = Token::lexer(input);
    while let Some(token) = lexer.next() {
        println!("{:?}", token);
    }
}
```

--------------------------------

### Rust Enum with Logos #[token] and #[regex] Attributes

Source: https://logos.maciej.codes/attributes/token_and_regex

Demonstrates the basic syntax for applying #[token] and #[regex] attributes to enum variants in Rust when using the Logos derive macro. It shows the required 'literal' parameter and optional parameters for callbacks, priorities, and ignore flags.

```rust
#![allow(unused)]
fn main() {
#[derive(Logos)]
enum Token {
    #[token(literal [, callback, priority = <integer>, ignore(<flag>, ...)]]
    #[regex(literal [, callback, priority = <integer>, ignore(<flag>, ...)]]
    SomeVariant,
}
}
```

--------------------------------

### Implement Variable Definition and String Interpolation Callbacks

Source: https://logos.maciej.codes/examples/string-interpolation

Three Rust callback functions for Logos lexer system that handle variable definitions with string interpolation. The `variable_definition` callback processes variable assignments, `evaluate_interpolation` handles `${...}` expressions, and `get_variable_value` retrieves variable values from a symbol table. Uses context switching between lexers via the morph method to handle nested interpolation scenarios.

```rust
fn get_string_content(lex: &mut Lexer<StringContext>) -> String {
    let mut s = String::new();
    while let Some(Ok(token)) = lex.next() {
        match token {
            StringContext::Content => s.push_str(lex.slice()),
            StringContext::DollarSign => s.push('$'),
            StringContext::InterpolationStart(value) => s.push_str(&value),
            StringContext::Quote => break,
        }
    }
    s
}

fn variable_definition(lex: &mut Lexer<VariableDefinitionContext>) -> Option<(String, String)> {
    let id = lex.slice().to_string();
    if let Some(Ok(VariableDefinitionContext::Equals)) = lex.next() {
        if let Some(Ok(VariableDefinitionContext::Quote)) = lex.next() {
            let mut lex2 = lex.clone().morph::<StringContext>();
            let value = get_string_content(&mut lex2);
            *lex = lex2.morph();
            lex.extras.insert(id.clone(), value.clone());
            return Some((id, value));
        }
    }
    None
}
```

```rust
fn evaluate_interpolation(lex: &mut Lexer<StringContext>) -> Option<String> {
    let mut lex2 = lex.clone().morph::<StringInterpolationContext>();
    let mut interpolation = String::new();
    while let Some(result) = lex2.next() {
        match result {
            Ok(token) => match token {
                StringInterpolationContext::Id(value) => interpolation.push_str(&value),
                StringInterpolationContext::Quote => {
                    *lex = lex2.morph();
                    interpolation.push_str(&get_string_content(lex));
                    lex2 = lex.clone().morph();
                }
                StringInterpolationContext::InterpolationEnd => break,
            },
            Err(()) => panic!("Interpolation error"),
        }
    }
    *lex = lex2.morph();
    Some(interpolation)
}
```

```rust
fn get_variable_value(lex: &mut Lexer<StringInterpolationContext>) -> Option<String> {
    if let Some(value) = lex.extras.get(lex.slice()) {
        return Some(value.clone());
    }
    None
}
```