mdstream

https://docs.rs/mdstream/0.2.0
Admin
mdstream is a Rust crate for incremental markdown parsing and transformation, enabling real-time...
Tokens:86,129
Snippets:687
Trust Score:9.1
Update:5 months ago
Show doc for...
Tokens:
Raw
### Example Usage of a Documented Function

Source: https://docs.rs/mdstream/0.2.0/scrape-examples-help.html

Create an example in your project that calls a documented item from your library. This example will be scraped and included in the documentation for the called item.

```rust
// examples/ex.rs
fn main() {
  a_crate::a_func();
}
```

--------------------------------

### FnBoundaryPlugin Implementation of start

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/boundary.rs.html

Implements the `start` method for `FnBoundaryPlugin`. If a `start` closure is provided, it is called with the starting line.

```rust
    fn start(&mut self, line: &str) {
        if let Some(f) = self.start.as_mut() {
            (f)(line);

```

--------------------------------

### start Method Signature

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/trait.BoundaryPlugin.html

Called exactly once when a block is determined to start at the given `line`. This method is responsible for initializing the plugin's state for the new block.

```rust
fn start(&mut self, line: &str);
```

--------------------------------

### FnBoundaryPlugin::with_start

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.FnBoundaryPlugin.html

Configures the start behavior of the FnBoundaryPlugin.

```APIDOC
## FnBoundaryPlugin::with_start

### Description
Sets a closure to be called when the current block is determined to start.

### Signature
```rust
pub fn with_start<S>(self, start: S) -> Self
where
    S: FnMut(&str) + Send + Sync + 'static,
```

### Parameters
* `start`: A mutable closure that takes a string slice and is called once when the block starts.
```

--------------------------------

### TagBoundaryPlugin start Method

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.TagBoundaryPlugin.html

Callback function executed once when a block is determined to start at the given line. It takes a mutable reference to self.

```rust
fn start(&mut self, _line: &str)
```

--------------------------------

### FenceBoundaryPlugin: Start Method

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/boundary.rs.html

Initializes the plugin's state when a potential fence start is detected. It records the length of the fence and sets a flag indicating the block has just started.

```rust
fn start(&mut self, line: &str) {
    let len = self.fence_len_at_start(line);
    if len >= self.min_len {
        self.opened_len = Some(len);
        self.just_started = true;
    } else {
        self.opened_len = None;
        self.just_started = false;
    }
}
```

--------------------------------

### Configure FnBoundaryPlugin Start Callback

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.FnBoundaryPlugin.html

Adds a closure to be called exactly once when a block is determined to start. This allows for custom initialization logic for a new block.

```rust
pub fn with_start<S>(self, start: S) -> Self
where S: FnMut(&str) + Send + Sync + 'static,
```

--------------------------------

### Pulldown Incremental Parsing Example

Source: https://docs.rs/mdstream/0.2.0/src/pulldown_incremental/pulldown_incremental.rs.html

This is the main example demonstrating incremental parsing with `mdstream` and the `pulldown-cmark` adapter. It requires the `pulldown` feature to be enabled. The example shows how to append chunks of Markdown, process updates, and inspect committed and pending blocks, including tracking link presence and handling reference definition invalidation.

```rust
#![allow(unused)]

1//! Incremental `pulldown-cmark` parsing with `mdstream`.
2//!
3//! Run:
4//!   cargo run --features pulldown --example pulldown_incremental
5
6#[cfg(not(feature = "pulldown"))]
7fn main() {
8    eprintln!("This example requires the `pulldown` feature.");
9    eprintln!("Run: cargo run --features pulldown --example pulldown_incremental");
10}
11
12#[cfg(feature = "pulldown")]
13fn main() {
14    use mdstream::adapters::pulldown::{PulldownAdapter, PulldownAdapterOptions};
15    use mdstream::{MdStream, Options, ReferenceDefinitionsMode};
16    use pulldown_cmark::{Event, Options as PulldownOptions, Tag};
17
18    // Optional: demonstrate invalidation when reference definitions arrive late.
19    let opts = Options {
20        reference_definitions: ReferenceDefinitionsMode::Invalidate,
21        ..Default::default()
22    };
23    let mut s = MdStream::new(opts)
24        .with_pending_transformer(mdstream::IncompleteLinkPlaceholderTransformer::default());
25    s.push_pending_transformer(mdstream::IncompleteImageDropTransformer::default());
26
27    let mut adapter = PulldownAdapter::new(PulldownAdapterOptions {
28        pulldown: PulldownOptions::ENABLE_TABLES | PulldownOptions::ENABLE_STRIKETHROUGH,
29        prefer_display_for_pending: true,
30    });
31
32    let chunks = [
33        "See a [ref].\n\n",
34        "Now define it:\n\n",
35        "[ref]: https://example.com\n\n",
36        "And some `code`.\n",
37    ];
38
39    let mut tracked: Option<mdstream::BlockId> = None;
40
41    for (i, chunk) in chunks.iter().enumerate() {
42        println!("\n== tick {i} ==");
43        let update = s.append(chunk);
44
45        if !update.invalidated.is_empty() {
46            println!("invalidated: {:?}", update.invalidated);
47        }
48
49        adapter.apply_update(&update);
50
51        for b in &update.committed {
52            println!("committed block id={} kind={:?}", b.id.0, b.kind);
53            let events = adapter.committed_events(b.id).unwrap_or(&[]);
54            println!("  events.len={}", events.len());
55            // Track the first committed block so we can show invalidation effects later.
56            if tracked.is_none() {
57                tracked = Some(b.id);
58            }
59        }
60
61        if let Some(id) = tracked {
62            let events = adapter.committed_events(id).unwrap_or(&[]);
63            let has_link = events
64                .iter()
65                .any(|e| matches!(e, Event::Start(Tag::Link { .. })))
66                ;
67            println!("tracked committed id={} has_link={}", id.0, has_link);
68        }
69
70        if let Some(p) = &update.pending {
71            let events = adapter.parse_pending(p);
72            let has_link = events
73                .iter()
74                .any(|e| matches!(e, Event::Start(Tag::Link { .. })))
75                ;
76            println!(
77                "pending block id={} kind={:?} events.len={} has_link={}",
78                p.id.0,
79                p.kind,
80                events.len(),
81                has_link
82            );
83        } else {
84            println!("pending: <none>");
85        }
86    }
87}

```

--------------------------------

### BoundaryPlugin::start

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.TagBoundaryPlugin.html

Called once when the current block is determined to start at the given line.

```APIDOC
## `impl BoundaryPlugin for TagBoundaryPlugin`

### `fn start(&mut self, _line: &str)`

Called exactly once when the current block is determined to start at `line`.
```

--------------------------------

### Check for List Item Start

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/stream.rs.html

Placeholder function to check if a line starts a list item. The actual implementation would involve checking for list markers like '-', '*', '+', or numbers followed by a period.

```Rust
fn is_list_item_start(line: &str) -> bool {

```

--------------------------------

### FnBoundaryPlugin::with_start (Sync)

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/boundary.rs.html

Adds an optional `start` closure to an existing `FnBoundaryPlugin`. This closure is called once when a block is determined to start.

```rust
#[cfg(feature = "sync")]
    pub fn with_start<S>(mut self, start: S) -> Self
    where
        S: FnMut(&str) + Send + Sync + 'static,
    {
        self.start = Some(Box::new(start));
        self
    }
```

--------------------------------

### Pulldown Incremental Adapter Setup

Source: https://docs.rs/mdstream/0.2.0/mdstream/stream/struct.MdStream.html

Demonstrates setting up MdStream with `PulldownAdapter` and custom transformers for incremental markdown processing. It configures options for reference definitions and includes transformers for incomplete links and images.

```rust
fn main() {
    use mdstream::adapters::pulldown::{PulldownAdapter, PulldownAdapterOptions};
    use mdstream::{MdStream, Options, ReferenceDefinitionsMode};
    use pulldown_cmark::{Event, Options as PulldownOptions, Tag};

    // Optional: demonstrate invalidation when reference definitions arrive late.
    let opts = Options {
        reference_definitions: ReferenceDefinitionsMode::Invalidate,
        ..Default::default()
    };
    let mut s = MdStream::new(opts)
        .with_pending_transformer(mdstream::IncompleteLinkPlaceholderTransformer::default());
    s.push_pending_transformer(mdstream::IncompleteImageDropTransformer::default());

```

--------------------------------

### Minimal MdStream Usage

Source: https://docs.rs/mdstream/0.2.0/mdstream/stream/struct.MdStream.html

A basic example showing how to initialize MdStream, append chunks of markdown, and process the resulting document state.

```rust
fn main() {
    let mut stream = MdStream::new(Options::default());
    let mut state = DocumentState::new();

    let chunks = [
        "# Title\n\n",
        "Hello **wor",
        "ld**.\n\n",
        "A list:\n",
        "- item 1\n",
        "- item 2\n",
    ];

    for (i, chunk) in chunks.iter().enumerate() {
        println!("\n== tick {i} ==");
        let applied = state.apply(stream.append(chunk));

        if applied.reset {
            println!("reset: true");
        }
        if !applied.invalidated.is_empty() {
            println!("invalidated: {:?}", applied.invalidated);
        }

        println!("committed={}", state.committed().len());
        if let Some(p) = state.pending() {
            println!(
                "pending id={} kind={:?} text={:?}",
                p.id.0,
                p.kind,
                p.display_or_raw()
            );
        } else {
            println!("pending: <none>");
        }
    }

    println!("\n== finalize ==");
    let applied = state.apply(stream.finalize());
    println!(
        "final reset={} invalidated={:?}",
        applied.reset,
        applied.invalidated
    );
    println!("final committed={}", state.committed().len());
}
```

--------------------------------

### FnBoundaryPlugin::with_start (Async)

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/boundary.rs.html

Adds an optional `start` closure to an existing `FnBoundaryPlugin`. This closure is called once when a block is determined to start.

```rust
#[cfg(not(feature = "sync"))]
    pub fn with_start<S>(mut self, start: S) -> Self
    where
        S: FnMut(&str) + Send + 'static,
    {
        self.start = Some(Box::new(start));
        self
    }
```

--------------------------------

### Stateful Transformer Example

Source: https://docs.rs/mdstream/0.2.0/mdstream/stream/struct.MdStream.html

Demonstrates using a stateful transformer to count and prefix seen chunks. The transformer is applied to pending chunks.

```rust
fn main() {
    let mut s = MdStream::new(Options::default());

    let mut seen = 0usize;
    s.push_pending_transformer(FnPendingTransformer(
        move |input: mdstream::PendingTransformInput<'_>| {
            seen += 1;
            Some(format!("[seen={seen}] {}", input.display))
        },
    ));

    for chunk in ["Hello", " ", "**wor", "ld"] {
        let u = s.append(chunk);
        if let Some(p) = u.pending {
            println!("pending: {}", p.display_or_raw());
        }
    }
}
```

--------------------------------

### FenceBoundaryPlugin start Method

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.FenceBoundaryPlugin.html

Called once when the current block is determined to start at the given line. This method is part of the BoundaryPlugin trait implementation.

```rust
fn start(&mut self, line: &str)
```

--------------------------------

### Check for Markdown List Item Start (Rust)

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/stream.rs.html

Determines if a line starts a Markdown list item, handling various markers like '-', '+', '*', and numbered lists. It checks for the marker followed by a space or tab.

```rust
fn is_list_item_start(line: &str) -> bool {
    let s = line.trim_start();
    if s.len() < 2 {
        return false;
    }
    let bytes = s.as_bytes();
    match bytes[0] {
        b'-' | b'+' | b'*' => bytes[1] == b' ' || bytes[1] == b'\t',
        b'0'..=b'9' => {
            let mut i = 0usize;
            while i < bytes.len() && bytes[i].is_ascii_digit() {
                i += 1;
            }
            if i == 0 || i + 1 >= bytes.len() {
                return false;
            }
            (bytes[i] == b'.' || bytes[i] == b')')
                && (bytes[i + 1] == b' ' || bytes[i + 1] == b'\t')
        }
        _ => false,
    }
}
```

--------------------------------

### BoundaryPlugin::matches_start

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.TagBoundaryPlugin.html

Checks if a given line can start a block managed by this plugin.

```APIDOC
## `impl BoundaryPlugin for TagBoundaryPlugin`

### `fn matches_start(&self, line: &str) -> bool`

Pure predicate: return `true` if `line` can start this custom block. Read more
```

--------------------------------

### FenceBoundaryPlugin matches_start Method

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.FenceBoundaryPlugin.html

Checks if a given line can start a custom block defined by this FenceBoundaryPlugin. It returns true if the line meets the criteria for starting a fence.

```rust
fn matches_start(&self, line: &str) -> bool
```

--------------------------------

### Detect Code Fence Start

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/stream.rs.html

Identifies the start of a code fence (like ``` or ~~~) after optional leading spaces. It returns the fence character and its length.

```Rust
fn fence_start(line: &str) -> Option<(char, usize)> {
    let mut s = line;
    let mut spaces = 0usize;
    while spaces < 3 && s.starts_with(' ') {
        s = &s[1..];
        spaces += 1;
    }
    let bytes = s.as_bytes();
    if bytes.len() < 3 {
        return None;
    }
    let ch = bytes[0] as char;
    if ch != '`' && ch != '~' {
        return None;
    }
    let mut len = 0usize;
    while len < bytes.len() && bytes[len] == bytes[0] {
        len += 1;
    }
    if len < 3 {
        return None;
    }
    Some((ch, len))
}
```

--------------------------------

### FnBoundaryPlugin::new

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.FnBoundaryPlugin.html

Creates a new FnBoundaryPlugin with provided closures for matching start lines and updating block content.

```APIDOC
## FnBoundaryPlugin::new

### Description
Creates a new `FnBoundaryPlugin` with closures for `matches_start` and `update`.

### Signature
```rust
pub fn new<M, U>(matches_start: M, update: U) -> Self
where
    M: Fn(&str) -> bool + Send + Sync + 'static,
    U: FnMut(&str) -> BoundaryUpdate + Send + Sync + 'static,
```

### Parameters
* `matches_start`: A closure that takes a string slice and returns `true` if the line can start a custom block.
* `update`: A mutable closure that takes a string slice and returns a `BoundaryUpdate`.
```

--------------------------------

### Incremental Markdown Processing with PulldownAdapter

Source: https://docs.rs/mdstream/0.2.0/mdstream/adapters/pulldown/struct.PulldownAdapter.html

This example shows how to use PulldownAdapter to process markdown chunks incrementally. It demonstrates appending chunks, applying updates, and inspecting committed and pending events, including tracking block invalidations and link presence. This is useful for real-time markdown rendering or editing scenarios.

```rust
fn main() {
    use mdstream::adapters::pulldown::{PulldownAdapter, PulldownAdapterOptions};
    use mdstream::{MdStream, Options, ReferenceDefinitionsMode};
    use pulldown_cmark::{Event, Options as PulldownOptions, Tag};

    // Optional: demonstrate invalidation when reference definitions arrive late.
    let opts = Options {
        reference_definitions: ReferenceDefinitionsMode::Invalidate,
        ..Default::default()
    };
    let mut s = MdStream::new(opts)
        .with_pending_transformer(mdstream::IncompleteLinkPlaceholderTransformer::default());
    s.push_pending_transformer(mdstream::IncompleteImageDropTransformer::default());

    let mut adapter = PulldownAdapter::new(PulldownAdapterOptions {
        pulldown: PulldownOptions::ENABLE_TABLES | PulldownOptions::ENABLE_STRIKETHROUGH,
        prefer_display_for_pending: true,
    });

    let chunks = [
        "See a [ref].\n\n",
        "Now define it:\n\n",
        "[ref]: https://example.com\n\n",
        "And some `code`.\n",
    ];

    let mut tracked: Option<mdstream::BlockId> = None;

    for (i, chunk) in chunks.iter().enumerate() {
        println!("\n== tick {i} ==");
        let update = s.append(chunk);

        if !update.invalidated.is_empty() {
            println!("invalidated: {:?}", update.invalidated);
        }

        adapter.apply_update(&update);

        for b in &update.committed {
            println!("committed block id={} kind={:?}", b.id.0, b.kind);
            let events = adapter.committed_events(b.id).unwrap_or(&[]);
            println!("  events.len={}", events.len());
            // Track the first committed block so we can show invalidation effects later.
            if tracked.is_none() {
                tracked = Some(b.id);
            }
        }

        if let Some(id) = tracked {
            let events = adapter.committed_events(id).unwrap_or(&[]);
            let has_link = events
                .iter()
                .any(|e| matches!(e, Event::Start(Tag::Link { .. })))
            ;
            println!("tracked committed id={} has_link={}", id.0, has_link);
        }

        if let Some(p) = &update.pending {
            let events = adapter.parse_pending(p);
            let has_link = events
                .iter()
                .any(|e| matches!(e, Event::Start(Tag::Link { .. })))
            ;
            println!(
                "pending block id={} kind={:?} events.len={} has_link={}",
                p.id.0,
                p.kind,
                events.len(),
                has_link
            );
        } else {
            println!("pending: <none>");
        }
    }
}

```

--------------------------------

### Minimal MdStream Usage with DocumentState

Source: https://docs.rs/mdstream/0.2.0/mdstream/types/struct.Block.html

Illustrates a minimal setup for MdStream and DocumentState, processing markdown chunks and displaying committed and pending block information.

```rust
8fn main() {
9    let mut stream = MdStream::new(Options::default());
10    let mut state = DocumentState::new();
11
12    let chunks = [
13        "# Title\n\n",
14        "Hello **wor",
15        "ld**.\n\n",
16        "A list:\n",
17        "- item 1\n",
18        "- item 2\n",
19    ];
20
21    for (i, chunk) in chunks.iter().enumerate() {
22        println!("\n== tick {i} ==");
23        let applied = state.apply(stream.append(chunk));
24
25        if applied.reset {
26            println!("reset: true");
27        }
28        if !applied.invalidated.is_empty() {
29            println!("invalidated: {:?}", applied.invalidated);
30        }
31
32        println!("committed={}", state.committed().len());
33        if let Some(p) = state.pending() {
34            println!(
35                "pending id={} kind={{:?}} text={{:?}}",
36                p.id.0,
37                p.kind,
38                p.display_or_raw()
39            );
40        } else {
41            println!("pending: <none>");
42        }
43    }
44
45    println!("\n== finalize ==");
46    let applied = state.apply(stream.finalize());
47    println!(
48        "final reset={{}} invalidated={{:?}}",
49        applied.reset,
50        applied.invalidated
51    );
52    println!("final committed={}", state.committed().len());
53}
```

--------------------------------

### Example Usage of TagBoundaryPlugin

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.TagBoundaryPlugin.html

Illustrates a conceptual usage of the TagBoundaryPlugin within an HTML-like tag structure.

```html
<thinking>
...
</thinking>
```

--------------------------------

### Example Usage of PendingBlockRef

Source: https://docs.rs/mdstream/0.2.0/mdstream/types/struct.PendingBlockRef.html

Demonstrates appending chunks to an MdStream and inspecting the pending block reference, including its display or raw content.

```rust
fn main() {
    let mut s = MdStream::new(Options::default());

    let chunks = [
        "# Title\n\n",
        "```rs\nfn main() {\n",
        "    println!(\"hi\");\n",
        "}\n",
    ];
    for (i, chunk) in chunks.iter().enumerate() {
        let u = s.append_ref(chunk);
        println!("== step {i} ==");
        println!("committed: {}", u.committed.len());
        if let Some(p) = u.pending {
            println!("pending kind={:?}", p.kind);
            println!("{}", p.display_or_raw());
        }
    }
}
```

--------------------------------

### Markdown Stream Analysis Example

Source: https://docs.rs/mdstream/0.2.0/mdstream/analyze/struct.BlockHintMeta.html

Demonstrates how to use BlockHintAnalyzer with an AnalyzedStream to process markdown chunks. It shows how to inspect committed and pending blocks, and how to interpret BlockHintMeta flags like likely_incomplete, display_transformed, unclosed_code_fence, and unbalanced_math.

```rust
fn main() {
    // For demo purposes, disable terminator link/image handling and enable the built-in
    // Streamdown-compatible pending transformers instead.
    let opts = Options {
        footnotes: FootnotesMode::SingleBlock,
        terminator: mdstream::pending::TerminatorOptions {
            links: false,
            images: false,
            ..Default::default()
        },
        ..Default::default()
    };

    // Chain analyzers: (code fence meta, pending hint meta)
    let analyzer = (CodeFenceAnalyzer, BlockHintAnalyzer);
    let mut s = AnalyzedStream::new(opts, analyzer);
    let mut state = DocumentState::new();
    s.inner_mut()
        .push_pending_transformer(IncompleteLinkPlaceholderTransformer::default());
    s.inner_mut()
        .push_pending_transformer(IncompleteImageDropTransformer::default());

    let chunks = [
        "# Streaming demo\n\n",
        "Normal text with **bold",
        " continued**.\n\n",
        "See [docs](",
        " and an image ![alt](",
        "...\n\n",
        "```mermaid\n",
        "graph TD;\nA-->B;\n",
        "```\n\n",
        "After code fence.\n",
        "\n$$\nE = mc^2\n",
        "$$\n",
    ];

    for (i, chunk) in chunks.iter().enumerate() {
        println!("\n== append step {i} ==");
        let u = s.append(chunk);
        let update = u.update;

        for (block, meta) in update
            .committed
            .iter()
            .zip(u.committed_meta.iter().map(|m| &m.meta))
        {
            print_block("committed", block.id.0, block.kind, &block.raw);

            if let Some(code_meta) = &meta.0 {
                match code_meta.class {
                    CodeFenceClass::Mermaid => println!("  meta: code fence class=mermaid"),
                    CodeFenceClass::Json => println!("  meta: code fence class=json"),
                    CodeFenceClass::Other => {}
                }
            }
        }

        if let Some(p) = &update.pending {
            print_block("pending  ", p.id.0, p.kind, &p.raw);

            if let Some(pm) = &u.pending_meta {
                let hint = pm.meta.1.unwrap_or(mdstream::BlockHintMeta { flags: 0 });
                if hint.likely_incomplete() {
                    let mut flags = Vec::new();
                    if hint.has(BlockHintMeta::DISPLAY_TRANSFORMED) {
                        flags.push("display_transformed");
                    }
                    if hint.has(BlockHintMeta::UNCLOSED_CODE_FENCE) {
                        flags.push("unclosed_code_fence");
                    }
                    if hint.has(BlockHintMeta::UNBALANCED_MATH) {
                        flags.push("unbalanced_math");
                    }
                    println!("  hint: likely_incomplete flags={flags:?}");
                }
            }

            if let Some(display) = &p.display {
                if display != &p.raw {
                    println!("  display (pending only):");
                    println!(display);
                }
            }
        } else {
            println!("pending: <none>");
        }

        let applied = state.apply(update);
        if applied.reset {
            println!("reset: true (drop cached UI state and rebuild)");
        }

        // State view (what a UI would keep).
        println!(
            "state: committed={} pending={}",
            state.committed().len(),
            state.pending().is_some()
        );
    }

    println!("\n== finalize ==");
    let u = s.finalize();
    let update = u.update;
    for b in &update.committed {
        print_block("committed", b.id.0, b.kind, &b.raw);
    }
    state.apply(update);
    println!("pending: {:?}", state.pending().map(|b| (b.id.0, b.kind)));
}

```

--------------------------------

### Incremental Parsing with PulldownAdapter

Source: https://docs.rs/mdstream/0.2.0/mdstream/adapters/pulldown/struct.PulldownAdapter.html

Demonstrates incremental Markdown parsing using PulldownAdapter. It shows how to append chunks of Markdown, apply updates, and retrieve parsed events for committed and pending blocks. This example highlights the effect of late-arriving reference definitions on parsing.

```rust
fn main() {
    use mdstream::adapters::pulldown::{PulldownAdapter, PulldownAdapterOptions};
    use mdstream::{MdStream, Options, ReferenceDefinitionsMode};
    use pulldown_cmark::{Event, Options as PulldownOptions, Tag};

    // Optional: demonstrate invalidation when reference definitions arrive late.
    let opts = Options {
        reference_definitions: ReferenceDefinitionsMode::Invalidate,
        ..Default::default()
    };
    let mut s = MdStream::new(opts)
        .with_pending_transformer(mdstream::IncompleteLinkPlaceholderTransformer::default());
    s.push_pending_transformer(mdstream::IncompleteImageDropTransformer::default());

    let mut adapter = PulldownAdapter::new(PulldownAdapterOptions {
        pulldown: PulldownOptions::ENABLE_TABLES | PulldownOptions::ENABLE_STRIKETHROUGH,
        prefer_display_for_pending: true,
    });

    let chunks = [
        "See a [ref].\n\n",
        "Now define it:\n\n",
        "[ref]: https://example.com\n\n",
        "And some `code`.\n",
    ];

    let mut tracked: Option<mdstream::BlockId> = None;

    for (i, chunk) in chunks.iter().enumerate() {
        println!("\n== tick {i} ==");
        let update = s.append(chunk);

        if !update.invalidated.is_empty() {
            println!("invalidated: {:?}", update.invalidated);
        }

        adapter.apply_update(&update);

        for b in &update.committed {
            println!("committed block id={} kind={:?}", b.id.0, b.kind);
            let events = adapter.committed_events(b.id).unwrap_or(&[]);
            println!("  events.len={}", events.len());
            // Track the first committed block so we can show invalidation effects later.
            if tracked.is_none() {
                tracked = Some(b.id);
            }
        }

        if let Some(id) = tracked {
            let events = adapter.committed_events(id).unwrap_or(&[]);
            let has_link = events
                .iter()
                .any(|e| matches!(e, Event::Start(Tag::Link { .. })));
            println!("tracked committed id={} has_link={}", id.0, has_link);
        }

        if let Some(p) = &update.pending {
            let events = adapter.parse_pending(p);
            let has_link = events
                .iter()
                .any(|e| matches!(e, Event::Start(Tag::Link { .. })));
            println!(
                "pending block id={} kind={:?} events.len={} has_link={}",
                p.id.0,
                p.kind,
                events.len(),
                has_link
            );
        } else {
            println!("pending: <none>");
        }
    }
}

```

--------------------------------

### Borrowed Updates Example

Source: https://docs.rs/mdstream/0.2.0/mdstream/stream/struct.MdStream.html

Shows how to append string slices using `append_ref` and observe the committed and pending states. This is useful when you want to avoid cloning strings.

```rust
fn main() {
    let mut s = MdStream::new(Options::default());

    let chunks = [
        "# Title\n\n",
        "```rs\nfn main() {\n",
        "    println!(\"hi\");\n",
        "}
",
    ];
    for (i, chunk) in chunks.iter().enumerate() {
        let u = s.append_ref(chunk);
        println!("== step {i} ==");
        println!("committed: {}", u.committed.len());
        if let Some(p) = u.pending {
            println!("pending kind={:?}", p.kind);
            println!("{}", p.display_or_raw());
        }
    }
}
```

--------------------------------

### Check for Markdown List Item Start Prefix (Rust)

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/stream.rs.html

A streaming-only heuristic to identify potential Markdown list item starters. This is useful for handling list markers split across input chunks, such as '-' followed by ' item'. It checks for prefixes like '-', '+', '*', or numbers followed by '.' or ')'.

```rust
fn is_list_item_start_prefix(line: &str) -> bool {
    // Streaming-only heuristic: treat certain "prefix" lines as potential list item starters.
    // This prevents premature block commits when the marker is split across chunks (e.g. "-" then " item").
    let s = line.trim_start();
    if s.is_empty() {
        return false;
    }
    let bytes = s.as_bytes();
    match bytes[0] {
        b'-' | b'+' | b'*' => s.len() == 1,
        b'0'..=b'9' => {
            let mut i = 0usize;
            while i < bytes.len() && bytes[i].is_ascii_digit() {
                i += 1;
            }
            if i == 0 {
                return false;
            }
            if i == bytes.len() {
                // Digits only: could become "1." / "1)" with more input.
                return true;
            }
            if bytes[i] != b'.' && bytes[i] != b')' {
                return false;
            }
            if i + 1 == bytes.len() {
                // "1." or "1)" without the required whitespace yet.
                return true;
            }
            false
        }
        _ => false,
    }
}
```

--------------------------------

### Default PulldownAdapterOptions

Source: https://docs.rs/mdstream/0.2.0/mdstream/adapters/pulldown/struct.PulldownAdapterOptions.html

Provides the default configuration for PulldownAdapterOptions. This is useful when you need a basic setup without custom modifications.

```rust
fn default() -> Self
```

--------------------------------

### Check for Blockquote Start

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/stream.rs.html

Determines if a line begins a blockquote by checking for a '>' character after trimming leading whitespace.

```Rust
fn is_blockquote_start(line: &str) -> bool {
    let trimmed = line.trim_start();
    trimmed.starts_with('>')
}
```

--------------------------------

### Minimal mdstream Example

Source: https://docs.rs/mdstream/0.2.0/src/minimal/minimal.rs.html

This snippet illustrates the core functionality of mdstream by processing text chunks and printing the state changes. It's useful for understanding the basic flow of appending data and observing the document state updates.

```Rust
use mdstream::{DocumentState, MdStream, Options};

fn main() {
    let mut stream = MdStream::new(Options::default());
    let mut state = DocumentState::new();

    let chunks = [
        "# Title\n\n",
        "Hello **wor",
        "ld**.\n\n",
        "A list:\n",
        "- item 1\n",
        "- item 2\n",
    ];

    for (i, chunk) in chunks.iter().enumerate() {
        println!("\n== tick {i} ==");
        let applied = state.apply(stream.append(chunk));

        if applied.reset {
            println!("reset: true");
        }
        if !applied.invalidated.is_empty() {
            println!("invalidated: {:?}", applied.invalidated);
        }

        println!("committed={}", state.committed().len());
        if let Some(p) = state.pending() {
            println!(
                "pending id={} kind={:?} text={:?}",
                p.id.0,
                p.kind,
                p.display_or_raw()
            );
        } else {
            println!("pending: <none>");
        }
    }

    println!("\n== finalize ==");
    let applied = state.apply(stream.finalize());
    println!(
        "final reset={} invalidated={:?}",
        applied.reset,
        applied.invalidated
    );
    println!("final committed={}", state.committed().len());
}
```

--------------------------------

### Get Display or Raw Content

Source: https://docs.rs/mdstream/0.2.0/mdstream/types/struct.Block.html

Retrieves the display string if available, otherwise returns the raw string. Useful for rendering pending blocks.

```rust
println!("pending: {}", p.display_or_raw());
```

--------------------------------

### Streaming Markdown Analysis with mdstream

Source: https://docs.rs/mdstream/0.2.0/src/tui_like/tui_like.rs.html

This example demonstrates how to use `AnalyzedStream` to process markdown chunks. It configures options to disable terminator handling and enable pending transformers, then chains analyzers for code fences and block hints. The output shows the processing of various markdown elements, including code blocks and math, and how the state is updated.

```rust
use mdstream::{
    AnalyzedStream,
    BlockHintAnalyzer,
    BlockHintMeta,
    BlockKind,
    CodeFenceAnalyzer,
    CodeFenceClass,
    DocumentState,
    FootnotesMode,
    IncompleteImageDropTransformer,
    IncompleteLinkPlaceholderTransformer,
    Options,
};

fn print_block(prefix: &str, id: u64, kind: BlockKind, text: &str) {
    let mut first = text.lines().next().unwrap_or("");
    if first.len() > 80 {
        first = &first[..80];
    }
    println!("{prefix} id={id} kind={kind:?} first_line={first:?}");
}

fn main() {
    // For demo purposes, disable terminator link/image handling and enable the built-in
    // Streamdown-compatible pending transformers instead.
    let opts = Options {
        footnotes: FootnotesMode::SingleBlock,
        terminator: mdstream::pending::TerminatorOptions {
            links: false,
            images: false,
            ..Default::default()
        },
        ..Default::default()
    };

    // Chain analyzers: (code fence meta, pending hint meta)
    let analyzer = (CodeFenceAnalyzer, BlockHintAnalyzer);
    let mut s = AnalyzedStream::new(opts, analyzer);
    let mut state = DocumentState::new();
    s.inner_mut()
        .push_pending_transformer(IncompleteLinkPlaceholderTransformer::default());
    s.inner_mut()
        .push_pending_transformer(IncompleteImageDropTransformer::default());

    let chunks = [
        "# Streaming demo\n\n",
        "Normal text with **bold",
        " continued**.\n\n",
        "See [docs](",
        " and an image ![alt](",
        "...\n\n",
        "```mermaid\n",
        "graph TD;\nA-->B;\n",
        "```\n\n",
        "After code fence.\n",
        "\n$$\nE = mc^2\n",
        "$$\n",
    ];

    for (i, chunk) in chunks.iter().enumerate() {
        println!("\n== append step {i} ==");
        let u = s.append(chunk);
        let update = u.update;

        for (block, meta) in update
            .committed
            .iter()
            .zip(u.committed_meta.iter().map(|m| &m.meta))
        {
            print_block("committed", block.id.0, block.kind, &block.raw);

            if let Some(code_meta) = &meta.0 {
                match code_meta.class {
                    CodeFenceClass::Mermaid => println!("  meta: code fence class=mermaid"),
                    CodeFenceClass::Json => println!("  meta: code fence class=json"),
                    CodeFenceClass::Other => {}
                }
            }
        }

        if let Some(p) = &update.pending {
            print_block("pending  ", p.id.0, p.kind, &p.raw);

            if let Some(pm) = &u.pending_meta {
                let hint = pm.meta.1.unwrap_or(mdstream::BlockHintMeta { flags: 0 });
                if hint.likely_incomplete() {
                    let mut flags = Vec::new();
                    if hint.has(BlockHintMeta::DISPLAY_TRANSFORMED) {
                        flags.push("display_transformed");
                    }
                    if hint.has(BlockHintMeta::UNCLOSED_CODE_FENCE) {
                        flags.push("unclosed_code_fence");
                    }
                    if hint.has(BlockHintMeta::UNBALANCED_MATH) {
                        flags.push("unbalanced_math");
                    }
                    println!("  hint: likely_incomplete flags={{flags:?}}");
                }
            }

            if let Some(display) = &p.display {
                if display != &p.raw {
                    println!("  display (pending only):");
                    println!("{display}");
                }
            }
        } else {
            println!("pending: <none>");
        }

        let applied = state.apply(update);
        if applied.reset {
            println!("reset: true (drop cached UI state and rebuild)");
        }

        // State view (what a UI would keep).
        println!(
            "state: committed={{state.committed().len()}} pending={{state.pending().is_some()}}"
        );
    }

    println!("\n== finalize ==");
    let u = s.finalize();
    let update = u.update;
    for b in &update.committed {
        print_block("committed", b.id.0, b.kind, &b.raw);
    }
    state.apply(update);
    println!("pending: {{state.pending().map(|b| (b.id.0, b.kind))}}");
}

```

--------------------------------

### Get Tail Window of Text

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/pending/terminator.rs.html

Extracts the last `window_bytes` of a given text. If the text is shorter than `window_bytes`, the entire text is returned. It ensures the returned slice starts on a valid UTF-8 character boundary.

```rust
fn tail_window(text: &str, window_bytes: usize) -> (&str, usize) {
    if text.len() <= window_bytes {
        return (text, 0);
    }
    let start = text.len() - window_bytes;
    // Move to char boundary.
    let mut s = start;
    while !text.is_char_boundary(s) {
        s += 1;
    }
    (&text[s..], s)
}
```

--------------------------------

### Initialize PulldownAdapter with Options

Source: https://docs.rs/mdstream/0.2.0/mdstream/stream/struct.MdStream.html

Demonstrates how to create a new PulldownAdapter instance with specific parsing options enabled, such as tables and strikethrough.

```rust
let mut adapter = PulldownAdapter::new(PulldownAdapterOptions {
        pulldown: PulldownOptions::ENABLE_TABLES | PulldownOptions::ENABLE_STRIKETHROUGH,
        prefer_display_for_pending: true,
    });
```

--------------------------------

### Check for Footnote Definition Start

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/stream/footnotes.rs.html

Use this function to determine if a line begins a footnote definition. It checks for the pattern `[^id]:` after trimming leading whitespace.

```rust
pub(super) fn is_footnote_definition_start(line: &str) -> bool {
    let s = line.trim_start();
    s.starts_with("[^") && s.contains("]:")
}
```

--------------------------------

### Create FnBoundaryPlugin with Closures

Source: https://docs.rs/mdstream/0.2.0/mdstream/boundary/struct.FnBoundaryPlugin.html

Constructs a new FnBoundaryPlugin instance. It requires closures for matching block start lines and updating block state. The closures must be thread-safe and have a static lifetime.

```rust
pub fn new<M, U>(matches_start: M, update: U) -> Self
where M: Fn(&str) -> bool + Send + Sync + 'static,
      U: FnMut(&str) -> BoundaryUpdate + Send + Sync + 'static,
```

--------------------------------

### Block Methods

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/types.rs.html

Provides utility methods for the Block struct, such as retrieving the displayable string and parsing code fence headers. Use `display_or_raw` to get the content to render.

```rust
impl Block {
    pub fn display_or_raw(&self) -> &str {
        self.display.as_deref().unwrap_or(&self.raw)
    }

    pub fn code_fence_header(&self) -> Option<crate::syntax::CodeFenceHeader<'_>> {
        if self.kind != BlockKind::CodeFence {
            return None;
        }
        crate::syntax::parse_code_fence_header_from_block(&self.raw)
    }

    pub fn code_fence_language(&self) -> Option<&str> {
        self.code_fence_header().and_then(|h| h.language)
    }
}
```

--------------------------------

### Stateful Pending Transformer Example

Source: https://docs.rs/mdstream/0.2.0/src/stateful_transformer/stateful_transformer.rs.html

This snippet demonstrates creating and using a stateful pending transformer. The transformer increments a counter for each input and formats the output with the current count. It's useful for scenarios where transformations need to track state across multiple inputs.

```rust
1//! Demonstrate a stateful pending transformer.
2//!
3//! Run:
4//!   cargo run --example stateful_transformer
5
6use mdstream::{FnPendingTransformer, MdStream, Options};
7
8fn main() {
9    let mut s = MdStream::new(Options::default());
10
11    let mut seen = 0usize;
12    s.push_pending_transformer(FnPendingTransformer(
13        move |input: mdstream::PendingTransformInput<'_>| {
14            seen += 1;
15            Some(format!("[seen={seen}] {}", input.display))
16        },
17    ));
18
19    for chunk in ["Hello", " ", "**wor", "ld"] {
20        let u = s.append(chunk);
21        if let Some(p) = u.pending {
22            println!("pending: {}", p.display_or_raw());
23        }
24    }
25}

```

--------------------------------

### PulldownAdapter::new

Source: https://docs.rs/mdstream/0.2.0/mdstream/adapters/pulldown/struct.PulldownAdapter.html

Constructs a new PulldownAdapter with the specified options. This is the primary way to initialize the adapter for processing Markdown.

```APIDOC
## PulldownAdapter::new

### Description
Constructs a new `PulldownAdapter` with the specified options.

### Method
```rust
pub fn new(opts: PulldownAdapterOptions) -> Self
```

### Parameters
* `opts` (`PulldownAdapterOptions`): Options to configure the pulldown parser and adapter behavior.
```

--------------------------------

### ContainerBoundaryPlugin: Helper for container name start character

Source: https://docs.rs/mdstream/0.2.0/src/mdstream/boundary.rs.html

Checks if a byte is a valid starting character for a container name (alphanumeric or underscore).

```rust
fn is_container_name_start(b: u8) -> bool {
    b.is_ascii_alphanumeric() || b == b'_'
}
```

--------------------------------

### Initialize and Process Markdown with AnalyzedStream

Source: https://docs.rs/mdstream/0.2.0/mdstream/analyze/struct.AnalyzedStream.html

Demonstrates initializing AnalyzedStream with custom options and analyzers, then processing markdown chunks. It shows how to append data incrementally and inspect the resulting committed and pending blocks, along with their associated metadata.

```rust
fn main() {
    // For demo purposes, disable terminator link/image handling and enable the built-in
    // Streamdown-compatible pending transformers instead.
    let opts = Options {
        footnotes: FootnotesMode::SingleBlock,
        terminator: mdstream::pending::TerminatorOptions {
            links: false,
            images: false,
            ..Default::default()
        },
        ..Default::default()
    };

    // Chain analyzers: (code fence meta, pending hint meta)
    let analyzer = (CodeFenceAnalyzer, BlockHintAnalyzer);
    let mut s = AnalyzedStream::new(opts, analyzer);
    let mut state = DocumentState::new();
    s.inner_mut()
        .push_pending_transformer(IncompleteLinkPlaceholderTransformer::default());
    s.inner_mut()
        .push_pending_transformer(IncompleteImageDropTransformer::default());

    let chunks = [
        "# Streaming demo\n\n",
        "Normal text with **bold",
        " continued**.\n\n",
        "See [docs](",
        " and an image ![alt](",
        "...\n\n",
        "```mermaid\n",
        "graph TD;\nA-->B;\n",
        "```\n\n",
        "After code fence.\n",
        "\n$$\nE = mc^2\n",
        "$$\n",
    ];

    for (i, chunk) in chunks.iter().enumerate() {
        println!("\n== append step {i} ==");
        let u = s.append(chunk);
        let update = u.update;

        for (block, meta) in update
            .committed
            .iter()
            .zip(u.committed_meta.iter().map(|m| &m.meta))
        {
            print_block("committed", block.id.0, block.kind, &block.raw);

            if let Some(code_meta) = &meta.0 {
                match code_meta.class {
                    CodeFenceClass::Mermaid => println!("  meta: code fence class=mermaid"),
                    CodeFenceClass::Json => println!("  meta: code fence class=json"),
                    CodeFenceClass::Other => {}
                }
            }
        }

        if let Some(p) = &update.pending {
            print_block("pending  ", p.id.0, p.kind, &p.raw);

            if let Some(pm) = &u.pending_meta {
                let hint = pm.meta.1.unwrap_or(mdstream::BlockHintMeta { flags: 0 });
                if hint.likely_incomplete() {
                    let mut flags = Vec::new();
                    if hint.has(BlockHintMeta::DISPLAY_TRANSFORMED) {
                        flags.push("display_transformed");
                    }
                    if hint.has(BlockHintMeta::UNCLOSED_CODE_FENCE) {
                        flags.push("unclosed_code_fence");
                    }
                    if hint.has(BlockHintMeta::UNBALANCED_MATH) {
                        flags.push("unbalanced_math");
                    }
                    println!("  hint: likely_incomplete flags={flags:?}");
                }
            }

            if let Some(display) = &p.display {
                if display != &p.raw {
                    println!("  display (pending only):");
                    println!(display);
                }
            }
        } else {
            println!("pending: <none>");
        }

        let applied = state.apply(update);
        if applied.reset {
            println!("reset: true (drop cached UI state and rebuild)");
        }

        // State view (what a UI would keep).
        println!(
            "state: committed={} pending={}",
            state.committed().len(),
            state.pending().is_some()
        );
    }

    println!("\n== finalize ==");
    let u = s.finalize();
    let update = u.update;
    for b in &update.committed {
        print_block("committed", b.id.0, b.kind, &b.raw);
    }
    state.apply(update);
    println!("pending: {:?}", state.pending().map(|b| (b.id.0, b.kind)));
}
```
Tokens:
Raw