Logos

https://logos.maciej.codes
Admin
Logos is a fast, easy-to-use lexer generator for Rust that provides a derive macro and a Lexer...
Tokens:35,497
Snippets:206
Trust Score:7.2
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install mdBook and serve the book

Source: https://logos.maciej.codes/contributing/setup.html

Installs mdBook and mdbook-linkcheck2, then builds and serves the book with live-reload.

```bash
cargo install mdbook
```

```bash
cargo install mdbook-linkcheck2
```

```bash
mdbook serve book --open
```

--------------------------------

### Run the string-interpolation example with cargo

Source: https://logos.maciej.codes/examples/string-interpolation.html

Command to run the example from the logos repository.

```bash
cargo run --example string-interpolation
```

--------------------------------

### Run the array language example

Source: https://logos.maciej.codes/examples/array-language.html

Runs the example program with the provided array program file.

```bash
cargo run --example array_language examples/array_program.txt
```

--------------------------------

### Run extras example with cargo run

Source: https://logos.maciej.codes/extras.html

Runs the extras example with Cargo.toml as the input file. Requires cloning the repository first.

```bash
cargo run --example extras Cargo.toml
```

--------------------------------

### Install nightly and build documentation

Source: https://logos.maciej.codes/contributing/setup.html

Installs the nightly Rust toolchain and builds the documentation with a configuration similar to docs.rs.

```bash
rustup install nightly
```

```bash
RUSTDOCFLAGS="--cfg docsrs" cargo +nightly doc \
    --features debug \
    -Zunstable-options \
    -Zrustdoc-scrape-examples \
    --no-deps \
    --open \
```

--------------------------------

### Run the json-borrowed example

Source: https://logos.maciej.codes/examples/json_borrowed.html

Run the borrowed-data JSON parser example. Requires cloning the repository first.

```bash
cargo run --example json-borrowed examples/example.json
```

--------------------------------

### Full extras example with main function

Source: https://logos.maciej.codes/extras.html

Complete runnable example for using Logos::Extras to track line and column positions. Reads a file from the command line, lexes it, and prints each word with its location.

```rust
use logos::{Lexer, Logos, Skip};
use std::env;
use std::fs;

/// Update the line count and the char index.
fn newline_callback(lex: &mut Lexer<Token>) -> Skip {
    lex.extras.0 += 1;
    lex.extras.1 = lex.span().end;
    Skip
}

/// Compute the line and column position for the current word.
fn word_callback(lex: &mut Lexer<Token>) -> (usize, usize) {
    let line = lex.extras.0;
    let column = lex.span().start - lex.extras.1;

    (line, column)
}

/// Simple tokens to retrieve words and their location.
#[derive(Debug, Logos)]
#[logos(extras = (usize, usize))]
#[logos(skip(r"\n", newline_callback))]
enum Token {
    #[regex(r"\w+", word_callback)]
    Word((usize, usize)),
}

fn main() {
    let src = fs::read_to_string(env::args().nth(1).expect("Expected file argument"))
        .expect("Failed to read file");

    let mut lex = Token::lexer(src.as_str());

    while let Some(token) = lex.next() {
        if let Ok(Token::Word((line, column))) = token {
            println!("Word '{}' found at ({}, {})", lex.slice(), line, column);
        }
    }
}
```

--------------------------------

### Full Rust code for json-borrowed example

Source: https://logos.maciej.codes/examples/json_borrowed.html

Full Rust source for the borrowed-data JSON parser example. It defines the Token and Value enums with borrowed string slices, and starts the recursive descent parser functions. The code is incomplete as shown; the source continues beyond the provided excerpt.

```rust
use logos::{Lexer, Logos, Span};

use std::collections::HashMap;
use std::env;
use std::fs;

type Error = (String, Span);

type Result<T> = std::result::Result<T, Error>;

/// All meaningful JSON tokens.
///
/// > NOTE: regexes for [`Token::Number`] and [`Token::String`] may not
/// > catch all possible values, especially for strings. If you find
/// > errors, please report them so that we can improve the regex.
#[derive(Debug, Logos)]
#[logos(skip r"[ \t\r\n\f]+")]
enum Token<'source> {
    #[token("false", |_| false)]
    #[token("true", |_| true)]
    Bool(bool),

    #[token("{")]
    BraceOpen,

    #[token("}")]
    BraceClose,

    #[token("[")]
    BracketOpen,

    #[token("]")]
    BracketClose,

    #[token(":")]
    Colon,

    #[token(",")]
    Comma,

    #[token("null")]
    Null,

    #[regex(r"-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?", |lex| lex.slice().parse::<f64>().unwrap())]
    Number(f64),

    #[regex(r#""([^"\\\x00-\x1F]|\\(["\\bnfrt/]|u[a-fA-F0-9]{4}))*""#, |lex| lex.slice())]
    String(&'source str),
}

/// Represent any valid JSON value.
#[allow(unused)]
#[derive(Debug)]
enum Value<'source> {
    /// null.
    Null,
    /// true or false.
    Bool(bool),
    /// Any floating point number.
    Number(f64),
    /// Any quoted string.
    String(&'source str),
    /// An array of values
    Array(Vec<Value<'source>>),
    /// An dictionary mapping keys and values.
    Object(HashMap<&'source str, Value<'source>>),
}

/// Parse a token stream into a JSON value.
fn parse_value<'source>(lexer: &mut Lexer<'source, Token<'source>>) -> Result<Value<'source>> {
    if let Some(token) = lexer.next() {
        match token {
            Ok(Token::Bool(b)) => Ok(Value::Bool(b)),
            Ok(Token::BraceOpen) => parse_object(lexer),
            Ok(Token::BracketOpen) => parse_array(lexer),
            Ok(Token::Null) => Ok(Value::Null),
            Ok(Token::Number(n)) => Ok(Value::Number(n)),
            Ok(Token::String(s)) => Ok(Value::String(s)),
            _ => Err((
                "unexpected token here (context: value)".to_owned(),
                lexer.span(),
            )),
        }
    } else {
        Err(("empty values are not allowed".to_owned(), lexer.span()))
    }
}

/// Parse a token stream into an array and return when
/// a valid terminator is found.
///
/// > NOTE: we assume '[' was consumed.
fn parse_array<'source>(lexer: &mut Lexer<'source, Token<'source>>) -> Result<Value<'source>> {
    let mut array = Vec::new();
    let span = lexer.span();
    let mut awaits_comma = false;
    let mut awaits_value = false;

```

--------------------------------

### Run Brainfuck interpreter example

Source: https://logos.maciej.codes/examples/brainfuck.html

Run the Brainfuck interpreter example on a Brainfuck source file. Requires cloning the repository first.

```bash
cargo run --example brainfuck examples/hello_world.bf
```

--------------------------------

### Run calculator example with cargo

Source: https://logos.maciej.codes/examples/calculator.html

Command to run the calculator example with a sample expression. The output shows the AST and the result -2.

```bash
cargo run --example calculator '1 + 7 * (3 - 4) / 2'
```

--------------------------------

### Output of calculator example

Source: https://logos.maciej.codes/examples/calculator.html

The output produced by the calculator example for the given expression. It shows the AST and the result -2.

```text
[AST]
Add(
    Int(
        1,
    ),
    Div(
        Mul(
            Int(
                7,
            ),
            Sub(
                Int(
                    3,
                ),
                Int(
                    4,
                ),
            ),
        ),
        Int(
            2,
        ),
    ),
)

[result]
-2
```

--------------------------------

### Run JSON parser example with cargo

Source: https://logos.maciej.codes/examples/json.html

Command to run the JSON parser example using cargo. Requires cloning the repository first.

```bash
cargo run --example json examples/example.json
```

--------------------------------

### Rust example: callbacks with closures and named functions

Source: https://logos.maciej.codes/callbacks.html

Shows how to attach callbacks to regex patterns using closure syntax or named functions. The callbacks can return Option or Result, and the example demonstrates parsing numbers with suffixes 'k' and 'm' into u64 values. The lexer processes the input "5 42k 75m" and asserts the resulting tokens and slices.

```rust
use logos::{Logos, Lexer};

// Note: callbacks can return `Option` or `Result`
fn kilo(lex: &mut Lexer<Token>) -> Option<u64> {
    let slice = lex.slice();
    let n: u64 = slice[..slice.len() - 1].parse().ok()?; // skip 'k'
    Some(n * 1_000)
}

fn mega(lex: &mut Lexer<Token>) -> Option<u64> {
    let slice = lex.slice();
    let n: u64 = slice[..slice.len() - 1].parse().ok()?; // skip 'm'
    Some(n * 1_000_000)
}

#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")]
enum Token {
    // Callbacks can use closure syntax, or refer
    // to a function defined elsewhere.
    //
    // Each pattern can have its own callback.
    #[regex("[0-9]+", |lex| lex.slice().parse().ok())]
    #[regex("[0-9]+k", kilo)]
    #[regex("[0-9]+m", mega)]
    Number(u64),
}

fn main() {
    let mut lex = Token::lexer("5 42k 75m");

    assert_eq!(lex.next(), Some(Ok(Token::Number(5))));
    assert_eq!(lex.slice(), "5");

    assert_eq!(lex.next(), Some(Ok(Token::Number(42_000))));
    assert_eq!(lex.slice(), "42k");

    assert_eq!(lex.next(), Some(Ok(Token::Number(75_000_000))));
    assert_eq!(lex.slice(), "75m");

    assert_eq!(lex.next(), None);
}
```

--------------------------------

### String Interpolation Example Program

Source: https://logos.maciej.codes/examples/string-interpolation.html

Example program demonstrating string interpolation with nested interpolation. The variables are assigned the values: name = Mark, greeting = Hi Mark!, surname = Scott, greeting2 = Hi Mark Scott!, and greeting3 = Hi Mark Scott!.

```text
name = 'Mark'
greeting = 'Hi ${name}!'
surname = 'Scott'
greeting2 = 'Hi ${name ' ' surname}!'
greeting3 = 'Hi ${name ' ${surname}!'}'
```

--------------------------------

### Define newline_callback and word_callback for extras

Source: https://logos.maciej.codes/extras.html

Callback functions for the extras example. newline_callback increments the line count and resets the char index; word_callback computes the line and column for the current word.

```rust
/// Update the line count and the char index.
fn newline_callback(lex: &mut Lexer<Token>) -> Skip {
    lex.extras.0 += 1;
    lex.extras.1 = lex.span().end;
    Skip
}

/// Compute the line and column position for the current word.
fn word_callback(lex: &mut Lexer<Token>) -> (usize, usize) {
    let line = lex.extras.0;
    let column = lex.span().start - lex.extras.1;

    (line, column)
}
```

--------------------------------

### Build fuzzing target with cargo afl build

Source: https://logos.maciej.codes/contributing/fuzzing.html

Builds the fuzzing target with cargo-afl. Run this from the fuzz directory before starting the fuzzing campaign.

```bash
cargo afl build
```

--------------------------------

### morph example: C-style language with Python blocks

Source: https://logos.maciej.codes/context-dependent-lexing.html

Shows how to use morph to switch from a C-style lexer to a Python lexer when encountering a Python block. The callback clones the original lexer, morphs it, collects tokens until ExitPythonBlock, then morphs back. Requires both lexers to share the same Source type and the Extras type to be convertible.

```rust
#![allow(unused)]
fn main() {
#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
enum CToken {
    /* Tokens supporting C syntax */
    // ...
    #[regex(r#"extern\s+"python"\s*\{"#, python_block_callback)]
    PythonBlock(Vec<PythonToken>),
}

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
enum PythonToken {
    #[token("}")]
    ExitPythonBlock,
    /* Tokens supporting Python syntax */
    // ...
}

fn python_block_callback(lex: &mut Lexer<CToken>) -> Option<Vec<PythonToken>> {
    let mut python_lexer = lex.clone().morph::<PythonToken>();
    let mut tokens = Vec::new();
    while let Some(token) = python_lexer.next() {
        match token {
            Ok(PythonToken::ExitPythonBlock) => break,
            Err(_) => return None,
            Ok(tok) => tokens.push(tok),
        }
    }
    *lex = python_lexer.morph();
    Some(tokens)
}
}
```

--------------------------------

### Run fuzzing campaign with cargo afl fuzz

Source: https://logos.maciej.codes/contributing/fuzzing.html

Starts the fuzzing campaign with cargo-afl. The -i and -o flags specify the input and output directories, and the target binary is ../target/debug/logos-fuzz. It may run for a very long time before finding a bug.

```bash
cargo afl fuzz -i in -o out ../target/debug/logos-fuzz
```

--------------------------------

### Example Logos lexer definition

Source: https://logos.maciej.codes/debugging.html

Defines a Token enum with literal and regex tokens, and a main function that lexes a sample input. Used to demonstrate debugging output.

```rust
use logos::Logos;

#[derive(Debug, Logos, PartialEq)]
enum Token {
    // Tokens can be literal strings, of any length.
    #[token("fast")]
    Fast,

    #[token(".")]
    Period,

    // Or regular expressions.
    #[regex("[a-zA-Z]+")]
    Text,
}
fn main() {
    let input = "Create ridiculously fast Lexers.";

    let mut lexer = Token::lexer(input);
    while let Some(token) = lexer.next() {
        println!("{:?}", token);
    }
}
```

--------------------------------

### Evaluate tokens to produce an array in Rust

Source: https://logos.maciej.codes/examples/array-language.html

Processes a sequence of tokens, updating an accumulator that starts empty. Number and array tokens append their contents, while sum and product operators combine all numbers into a singleton. The final accumulator is returned after all tokens are evaluated.

```rust
/// Evaluate a sequence of tokens to produce an array.
fn evaluate(tokens: &[Token]) -> Vec<i128> {
    let mut accumulator = Vec::new();
    for tok in tokens {
        match *tok {
            Token::Number(n) => accumulator.push(n),
            Token::Array(arr) => accumulator.extend(arr),
            Token::Product => {
                let n = accumulator.drain(..).product();
                accumulator.push(n);
            }
            Token::Sum => {
                let n = accumulator.drain(..).sum();
                accumulator.push(n);
            }
            Token::Reverse => accumulator.reverse(),
        }
    }
    accumulator
}
```

--------------------------------

### Run tests and benchmarks

Source: https://logos.maciej.codes/contributing/setup.html

Runs the test suite and benchmarks for the workspace.

```bash
cargo test --workspace
```

```bash
cargo bench --workspace --benches
```

--------------------------------

### Implement main function

Source: https://logos.maciej.codes/examples/calculator.html

Main function that reads an expression from the command line, lexes it, parses it into an AST, and evaluates it. Prints the AST and result, or error messages for lexer/parser failures.

```rust
fn main() {
    //reads the input expression from the command line
    let input = env::args()
        .nth(1)
        .expect("Expected expression argument (e.g. `1 + 7 * (3 - 4) / 5`)");

    //creates a lexer instance from the input
    let lexer = Token::lexer(&input);

    //splits the input into tokens, using the lexer
    let mut tokens = vec![];
    for (token, span) in lexer.spanned() {
        match token {
            Ok(token) => tokens.push(token),
            Err(e) => {
                println!("lexer error at {:?}: {}", span, e);
                return;
            }
        }
    }

    //parses the tokens to construct an AST
    let ast = match parser().parse(&tokens).into_result() {
        Ok(expr) => {
            println!("[AST]\n{:#?}", expr);
            expr
        }
        Err(e) => {
            println!("parse error: {:#?}", e);
            return;
        }
    };

    //evaluates the AST to get the result
    println!("\n[result]\n{}", ast.eval());
}
```

--------------------------------

### Format and lint with cargo fmt and cargo clippy

Source: https://logos.maciej.codes/contributing/setup.html

Formats the code and runs Rust's linter. Run these frequently to avoid failing automated tests.

```bash
cargo fmt
```

```bash
cargo clippy
```

--------------------------------

### Tailcall codegen per-state functions

Source: https://logos.maciej.codes/state-machine-codegen.html

Shows the tailcall code generation approach: a function per state, with transitions implemented by calling the next state's function. This is the default and generates faster code, but can overflow the stack until Rust gets guaranteed tail calls.

```rust
fn state0(lexer: Lexer, context: Context) -> Token {
    match lexer.read() {
        'a' => state1(lexer, context),
        'b' => state2(lexer, context),
        _ => Token::Error,
    }
}

// Etc ...
```

--------------------------------

### Lex a string with Logos::lexer and inspect tokens

Source: https://logos.maciej.codes/getting-started.html

Use Logos::lexer to turn a &str into an iterator of tokens. Each item is a Result<Token, _>; lex.span() and lex.slice() give the matched range and text. The example asserts the token sequence for "Create ridiculously fast Lexers.".

```rust
#![allow(unused)]
fn main() {
let mut lex = Token::lexer("Create ridiculously fast Lexers.");

assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.span(), 0..6);
assert_eq!(lex.slice(), "Create");

assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.span(), 7..19);
assert_eq!(lex.slice(), "ridiculously");

assert_eq!(lex.next(), Some(Ok(Token::Fast)));
assert_eq!(lex.span(), 20..24);
assert_eq!(lex.slice(), "fast");

assert_eq!(lex.next(), Some(Ok(Token::Text)));
assert_eq!(lex.slice(), "Lexers");
assert_eq!(lex.span(), 25..31);

assert_eq!(lex.next(), Some(Ok(Token::Period)));
assert_eq!(lex.span(), 31..32);
assert_eq!(lex.slice(), ".");

assert_eq!(lex.next(), None);
}
```

--------------------------------

### Lexing partial input with Lexer::new_partial

Source: https://logos.maciej.codes/partial.html

Shows how to use Lexer::new_partial to lex a buffer that is not yet complete, repeatedly extending the buffer until EOF, then switching to Lexer::new for the final tokens. Useful for streaming input from Read or AsyncRead without loading everything into memory.

```rust
while !is_eof { // We run while we still have unread data
    let mut lexer = Lexer::new_partial(buffer);
    while let Some(token) = lexer.next() {
        // Do something with the token
    }
    // We add more data to the buffer
    extend_buffer_with_more_data(&mut buffer);
}
// We lex the last tokens with the usual lexer because the buffer is now filled to the end
let mut lexer = Lexer::new(buffer);
while let Some(token) = lexer.next() {
    // Do something with the token
}
```

--------------------------------

### Build and print an error report with a label

Source: https://logos.maciej.codes/examples/json.html

Builds an error report with a label, then prints it to stderr. The report is created with a line number (12) and a message, and the label includes a span and a color. The eprint call outputs the report and unwrap is used to handle any errors.

```rust
Report::build(ReportKind::Error, &filename, 12)
    .with_message("Invalid JSON".to_string())
    .with_label(
        Label::new((&filename, span))
            .with_message(msg)
            .with_color(a),
    )
    .finish()
    .eprint((&filename, Source::from(src)))
    .unwrap();
```

--------------------------------

### Explicit source lifetime with #[logos(lifetime = 's)]

Source: https://logos.maciej.codes/attributes/logos.html

Use this example to understand how to specify an explicit source lifetime with the `#[logos(lifetime = 's)]` attribute. The token type has two lifetimes, so the source lifetime must be set explicitly. Token lifetimes are not set to `'s` when using an explicit source lifetime.

```rust
#[derive(Logos)]
#[logos(lifetime = 's)]
enum Foo<'s, 'a> {
    #[token("bar", |lex| lex.slice())]
    Bar(&'s str),
    #[token("baz", |_| "static")]
    Baz(&'a str),
}
```

--------------------------------

### Clone the Logos repository

Source: https://logos.maciej.codes/contributing/setup.html

Clones the Logos repository. If you have a fork, clone it instead.

```bash
git clone https://github.com/maciejhirsz/logos.git
```

--------------------------------

### Diff to switch JSON parser to borrowed data

Source: https://logos.maciej.codes/examples/json_borrowed.html

Diff showing the exact lines to change from the previous owned-data example to use borrowed string slices. It updates the Token and Value enums, the String token regex callback, and the parser function signatures to use &'source str and lifetime parameters.

```diff
_
@ 33c29
- enum Token {
+ enum Token<'source> {
@ 62,63c58,59
-     #[regex(r#""([^"\\\x00-\x1F]|\\(["\\bnfrt/]|u[a-fA-F0-9]{4}))*""#, |lex| lex.slice().to_owned())]
-     String(String),
+     #[regex(r#""([^"\\\x00-\x1F]|\\(["\\bnfrt/]|u[a-fA-F0-9]{4}))*""#, |lex| lex.slice())]
+     String(&'source str),
@ 70c66
- enum Value {
+ enum Value<'source> {
@ 78c74
-     String(String),
+     String(&'source str),
@ 80c76
-     Array(Vec<Value>),
+     Array(Vec<Value<'source>>),
@ 82c78
-     Object(HashMap<String, Value>),
+     Object(HashMap<&'source str, Value<'source>>),
@ 88c84
- fn parse_value<'source>(lexer: &mut Lexer<'source, Token>) -> Result<Value> {
+ fn parse_value<'source>(lexer: &mut Lexer<'source, Token<'source>>) -> Result<Value<'source>> {
@ 113c109
- fn parse_array<'source>(lexer: &mut Lexer<'source, Token>) -> Result<Value> {
+ fn parse_array<'source>(lexer: &mut Lexer<'source, Token<'source>>) -> Result<Value<'source>> {
@ 167c163
- fn parse_object<'source>(lexer: &mut Lexer<'source, Token>) -> Result<Value> {
+ fn parse_object<'source>(lexer: &mut Lexer<'source, Token<'source>>) -> Result<Value<'source>> {

```

--------------------------------

### Rust: String Interpolation Lexer Implementation

Source: https://logos.maciej.codes/examples/string-interpolation.html

Defines lexer contexts and functions to parse variable definitions and string interpolation using Logos. Includes a main function that tests the lexer with sample inputs.

```rust
use std::collections::HashMap;

use logos::{Lexer, Logos};

type SymbolTable = HashMap<String, String>;

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
#[logos(extras = SymbolTable)]
enum VariableDefinitionContext {
    #[regex(r"[[:alpha:]][[:alnum:]]*", variable_definition)]
    Id((String /* variable name */, String /* value */)),
    #[token("=")]
    Equals,
    #[token("'")]
    Quote,
}

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(extras = SymbolTable)]
enum StringContext {
    #[token("'")]
    Quote,
    #[regex("[^'$]+")]
    Content,
    #[token("${", evaluate_interpolation)]
    InterpolationStart(String /* evaluated value of the interpolation */),
    #[token("$")]
    DollarSign,
}

#[derive(Logos, Debug, PartialEq, Clone)]
#[logos(skip r"\s+")]
#[logos(extras = SymbolTable)]
enum StringInterpolationContext {
    #[regex(r"[[:alpha:]][[:alnum:]]*", get_variable_value)]
    Id(String /* value for the given id */),
    #[token("'")]
    Quote,
    #[token("}")]
    InterpolationEnd,
}

fn get_string_content(lex: &mut Lexer<StringContext>) -> String {
    let mut s = String::new();
    while let Some(Ok(token)) = lex.next() {
        match token {
            StringContext::Content => s.push_str(lex.slice()),
            StringContext::DollarSign => s.push('$'),
            StringContext::InterpolationStart(value) => s.push_str(&value),
            StringContext::Quote => break,
        }
    }
    s
}

fn variable_definition(lex: &mut Lexer<VariableDefinitionContext>) -> Option<(String, String)> {
    let id = lex.slice().to_string();
    if let Some(Ok(VariableDefinitionContext::Equals)) = lex.next() {
        if let Some(Ok(VariableDefinitionContext::Quote)) = lex.next() {
            let mut lex2 = lex.clone().morph::<StringContext>();
            let value = get_string_content(&mut lex2);
            *lex = lex2.morph();
            lex.extras.insert(id.clone(), value.clone());
            return Some((id, value));
        }
    }
    None
}

fn evaluate_interpolation(lex: &mut Lexer<StringContext>) -> Option<String> {
    let mut lex2 = lex.clone().morph::<StringInterpolationContext>();
    let mut interpolation = String::new();
    while let Some(result) = lex2.next() {
        match result {
            Ok(token) => match token {
                StringInterpolationContext::Id(value) => interpolation.push_str(&value),
                StringInterpolationContext::Quote => {
                    *lex = lex2.morph();
                    interpolation.push_str(&get_string_content(lex));
                    lex2 = lex.clone().morph();
                }
                StringInterpolationContext::InterpolationEnd => break,
            },
            Err(()) => panic!("Interpolation error"),
        }
    }
    *lex = lex2.morph();
    Some(interpolation)
}

fn get_variable_value(lex: &mut Lexer<StringInterpolationContext>) -> Option<String> {
    if let Some(value) = lex.extras.get(lex.slice()) {
        return Some(value.clone());
    }
    None
}

fn test_variable_definition(
    expected_id: &str,
    expected_value: &str,
    token: Option<Result<VariableDefinitionContext, ()>>,
) {
    if let Some(Ok(VariableDefinitionContext::Id((id, value)))) = token {
        assert_eq!(id, expected_id);
        assert_eq!(value, expected_value);
    } else {
        panic!("Expected key: {} not found", expected_id);
    }
}

fn main() {
    let mut lex = VariableDefinitionContext::lexer(
        "\
        name = 'Mark'\n\
        greeting = 'Hi ${name}!\n\
        surname = 'Scott'\n\
        greeting2 = 'Hi ${name ' ' surname}!\n\
        greeting3 = 'Hi ${name ' ${surname}!'}!\n\
        ",
    );
    test_variable_definition("name", "Mark", lex.next());
    test_variable_definition("greeting", "Hi Mark!", lex.next());
    test_variable_definition("surname", "Scott", lex.next());
    test_variable_definition("greeting2", "Hi Mark Scott!", lex.next());
    test_variable_definition("greeting3", "Hi Mark Scott!!", lex.next());
}
```

--------------------------------

### Custom error type with #[logos(error)]

Source: https://logos.maciej.codes/attributes/logos.html

Defines a custom error type for the Logos lexer. The error type must implement Clone, PartialEq, Default, and From<E> for each callback's error type. The example shows how to handle overflow and non-ASCII characters, and uses the alternate syntax #[logos(error(LexingError, LexingError::from_lexer))] to provide a callback that extracts the offending character from the lexer.

```rust
use logos::Logos;

use std::num::ParseIntError;

#[derive(Default, Debug, Clone, PartialEq)]
enum LexingError {
    InvalidInteger(String),
    NonAsciiCharacter(char),
    #[default]
    Other,
}

/// Error type returned by calling `lex.slice().parse()` to u8.
impl From<ParseIntError> for LexingError {
    fn from(err: ParseIntError) -> Self {
        use std::num::IntErrorKind::*;
        match err.kind() {
            PosOverflow | NegOverflow => LexingError::InvalidInteger("overflow error".to_owned()),
            _ => LexingError::InvalidInteger("other error".to_owned()),
        }
    }
}

impl LexingError {
    fn from_lexer(lex: &mut logos::Lexer<'_, Token>) -> Self {
        LexingError::NonAsciiCharacter(lex.slice().chars().next().unwrap())
    }
}

#[derive(Debug, Logos, PartialEq)]
#[logos(error(LexingError, LexingError::from_lexer))]
#[logos(skip r"[ \t]+")]
enum Token {
    #[regex(r"[a-zA-Z]+")]
    Word,
    #[regex(r"[0-9]+", |lex| lex.slice().parse())]
    Integer(u8),
}

fn main() {
    // 256 overflows u8, since u8's max value is 255.
    // 'é' is not a valid ASCII letter.
    let mut lex = Token::lexer("Hello 256 Jérome");

    assert_eq!(lex.next(), Some(Ok(Token::Word)));
    assert_eq!(lex.slice(), "Hello");

    assert_eq!(
        lex.next(),
        Some(Err(LexingError::InvalidInteger(
            "overflow error".to_owned()
        )))
    );
    assert_eq!(lex.slice(), "256");

    assert_eq!(lex.next(), Some(Ok(Token::Word)));
    assert_eq!(lex.slice(), "J");

    assert_eq!(lex.next(), Some(Err(LexingError::NonAsciiCharacter('é'))));
    assert_eq!(lex.slice(), "é");

    assert_eq!(lex.next(), Some(Ok(Token::Word)));
    assert_eq!(lex.slice(), "rome");

    assert_eq!(lex.next(), None);
}
```
Tokens:
Raw