Decruft

https://docs.rs/decruft/0.1.4/
Admin
Rust library that extracts clean, readable content from noisy HTML pages by removing ads,...
Tokens:353,310
Snippets:2,584
Trust Score:9.1
Update:4 months ago
Show doc for...
Tokens:
Raw
### Parse HTML with Default Options

Source: https://docs.rs/decruft/0.1.4/src/decruft/lib.rs.html

Demonstrates how to use the `parse` function with default options to extract content from an HTML string. This is a quick way to get started with the library.

```rust
#![deny(
    clippy::unwrap_used,
    clippy::panic,
    clippy::dbg_macro,
    clippy::todo,
    clippy::print_stdout,
    clippy::print_stderr
)]

//! # decruft
//! 
//! Extract clean, readable content from web pages.
//! 
//! Given a noisy HTML page (ads, navigation, sidebars, popups, cookie banners...),
//! decruft extracts the main content and metadata.
//! 
//! ## Quick start
//! 
//! ```
//! use decruft::{parse, DecruftOptions};
//! 
//! let html = r#"<html>
//!   <head><title>My Post - Blog</title></head>
//!   <body>
//!     <nav><a href="/">Home</a></nav>
//!     <article><h1>My Post</h1><p>The content.</p></article>
//!     <footer>Copyright 2025</footer>
//!   </body>
//! </html>"#;
//! 
//! let result = parse(html, &DecruftOptions::default());
//! assert!(result.content.contains("The content."));
//! assert!(!result.content.contains("Copyright"));
//! ```
//! 
//! Or even simpler with [`parse_with_defaults`]:
//! 
//! ```
//! let html = "<html><body><article><p>Hello</p></article></body></html>";
//! let result = decruft::parse_with_defaults(html);
//! assert!(result.content.contains("Hello"));
//! ```
```

--------------------------------

### General Search Examples

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/twitter.rs.html?search=

Illustrative examples of search queries for common Rust patterns and types.

```rust
std::vec
```

```rust
u32 -> bool
```

```rust
Option<T>, (T -> U) -> Option<U>
```

--------------------------------

### Example Search Queries

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html?search=

Demonstrates common search query patterns.

```text
* std::vec
```

```text
* u32 -> bool
```

```text
* Option<T>, (T -> U) -> Option<U>
```

--------------------------------

### Example Search Query

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/twitter.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Demonstrates a typical search query format for finding Rust function signatures.

```text
Option<T>, (T -> U) -> Option<U>
```

--------------------------------

### Extract Domain from URL

Source: https://docs.rs/decruft/0.1.4/src/decruft/metadata.rs.html?search=

This example shows how to extract the domain name (e.g., 'example.com') from a given URL.

```rust
let doc = Html::parse_document("<html><body></body></html>");
        let m = extract_metadata(&doc, Some("https://example.com/page"), None);
        assert_eq!(m.domain.as_deref(), Some("example.com"));
```

--------------------------------

### Test get() Returns None for 404 Response

Source: https://docs.rs/decruft/0.1.4/src/decruft/http.rs.html

Verifies that the `get` function returns `None` when a URL returns a 404 status code. This test is designed to not fail if the network is unavailable.

```Rust
#[test]
fn get_returns_none_for_404() {
    let result = get("https://httpbin.org/status/404");
    // Should be None regardless of whether ureq treats 404 as error
    // or our explicit check catches it
    if result.is_some() {
        // Network might be unavailable, or httpbin might behave
        // unexpectedly — don't fail
    }
}
```

--------------------------------

### Rust Example Usage of parse Function

Source: https://docs.rs/decruft/0.1.4/decruft/fn.parse.html?search=

Demonstrates how to use the `parse` function with a simple HTML string and default options. Asserts that the extracted content contains the expected text.

```rust
use decruft::{parse, DecruftOptions};

let html = "<html><body><article><p>Hello world</p></article></body></html>";
let result = parse(html, &DecruftOptions::default());
assert!(result.content.contains("Hello world"));
```

--------------------------------

### Parse RC Calls - Basic Example

Source: https://docs.rs/decruft/0.1.4/src/decruft/streaming_ssr.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Parses a string to extract RC call arguments. Useful for identifying template and content IDs.

```rust
let text = r#"$RC("B:0","S:0")"#;
let calls = parse_rc_calls(text);
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].0, "B:0");
assert_eq!(calls[0].1, "S:0");
```

--------------------------------

### Extract Basic Question Data from HTML

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/stackoverflow.rs.html?search=

Demonstrates extracting basic question data, such as vote count, body, and author, from simple HTML structures. This is a simplified example for testing purposes.

```Rust
fn extract_from_html_basic() {
    let html_str = r#"\n        <html>\n        <body>\n        <div id=\"question-header\"><h1 >How to foo?</h1></div>\n        <div class=\"question\">\n            <div class=\"js-vote-count\">42</div>\n            <div class=\"js-post-body\">\n                <p>I want to know how to foo.</p>\n            </div>\n            <div class=\"user-details\">\n                <a href=\"/users/1/alice\">Alice</a>\n            </div>\n        </div>\n        <div class=\"answer accepted-answer\">\n            <div class=\"js-vote-count\">10</div>\n            <div class=\"js-post-body\">\n
```

--------------------------------

### Parse HTML with Custom Options

Source: https://docs.rs/decruft/0.1.4/index.html

Use the `parse` function with `DecruftOptions` to extract content from HTML. This example demonstrates basic usage and assertions to verify content extraction.

```rust
use decruft::{parse, DecruftOptions};

let html = r#"<html>
  <head><title>My Post - Blog</title></head>
  <body>
    <nav><a href="/">Home</a></nav>
    <article><h1>My Post</h1><p>The content.</p></article>
    <footer>Copyright 2025</footer>
  </body>
</html>"#;

let result = parse(html, &DecruftOptions::default());
assert!(result.content.contains("The content."));
assert!(!result.content.contains("Copyright"));
```

--------------------------------

### get

Source: https://docs.rs/decruft/0.1.4/src/decruft/http.rs.html?search=

Fetches a URL using API defaults (10s timeout, `decruft/0.1` UA). Returns `None` on network errors, non-200 status, timeout, or body read failures.

```APIDOC
## get

### Description
Fetches a URL using API defaults (10s timeout, `decruft/0.1` UA). Returns `None` on network errors, non-200 status, timeout, or body read failures.

### Method
GET

### Endpoint
`{url}`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

### Request Example
```http
GET {url} HTTP/1.1
User-Agent: decruft/0.1
Host: example.com

```

### Response
#### Success Response (200)
- **body** (string) - The response body as a string.

#### Response Example
```json
"response body"
```

ERROR HANDLING:
- Returns `None` on transport errors, non-200 status codes, or read failures.
```

--------------------------------

### Get All Descendant Element IDs

Source: https://docs.rs/decruft/0.1.4/src/decruft/dom.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Retrieves a vector of all descendant element IDs starting from a given node ID.

```rust
pub fn all_descendant_elements(html: &Html, node_id: NodeId) -> Vec<NodeId> {
    let mut result = Vec::new();
    collect_all_descendants(html, node_id, &mut result);
    result
}
```

--------------------------------

### Collect Aside Footnotes

Source: https://docs.rs/decruft/0.1.4/src/decruft/footnotes.rs.html?search=

Collects footnotes from `<aside>` elements containing an `<ol>` with a `start` attribute. It parses the `start` attribute to determine the footnote number and extracts content, handling single or multiple list items.

```Rust
fn collect_aside_footnotes(
    html: &Html,
    main_content: NodeId,
    footnotes: &mut BTreeMap<usize, FootnoteData>,
    containers: &mut Vec<NodeId>,
) {
    let ols = dom::select_within(html, main_content, "aside > ol[start]");
    for ol_id in ols {
        let start_attr = dom::get_attr(html, ol_id, "start").unwrap_or_default();
        let Ok(start_num) = start_attr.parse::<usize>() else {
            continue;
        };
        if start_num < 1 {
            continue;
        }

        let items = dom::select_within(html, ol_id, "li");
        if items.is_empty() {
            continue;
        }

        let content = if items.len() == 1 {
            dom::inner_html(html, items[0])
        } else {
            let mut combined = String::new();
            for item in &items {
                combined.push_str("<p>");
                combined.push_str(&dom::inner_html(html, *item));
                combined.push_str("</p>");
            }
            combined
        };

        footnotes.insert(
            start_num,
            FootnoteData {
                content_html: strip_backrefs_from_html(&content),
                original_id: start_num.to_string(),
            },
        );

        // Remove the <aside> parent
        if let Some(aside_id) = dom::parent_element(html, ol_id) {
            containers.push(aside_id);
        }
    }
}
```

--------------------------------

### Build Content HTML

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/comments.rs.html?search=

Constructs the full HTML for a post, optionally including a comments section. Returns only the post section if comments are empty.

```rust
/// Build full content HTML for a post with optional comments section.
#[must_use]
pub fn build_content_html(site: &str, post_content: &str, comments: &str) -> String {
    let post_section = format!(
        "<div class=\"{site}\" post"><div class=\"post-content\">{post_content}</div></div>"
    );
    if comments.is_empty() {
        return post_section;
    }
    format!("{post_section}<hr><div class=\"{site}\" comments><h2>Comments</h2>{comments}</div>")
}
```

--------------------------------

### Any Trait Implementation

Source: https://docs.rs/decruft/0.1.4/decruft/struct.DecruftResult.html?search=u32+-%3E+bool

Provides a way to get the TypeId of a DecruftResult instance.

```rust
impl<T> Any for T
where T: 'static + ?Sized,

fn type_id(&self) -> TypeId

```

--------------------------------

### Custom HTML to Markdown Conversion with Handlers

Source: https://docs.rs/decruft/0.1.4/src/decruft/decruft.rs.html

Demonstrates how to build a custom HtmlToMarkdown converter by adding handlers for specific HTML elements like 'iframe'. This example shows how to extract the 'src' attribute and convert an iframe to a Markdown image link, with special handling for YouTube embeds.

```rust
use htmd::element_handler::HandlerResult;
use htmd::element_handler::Handlers;

HtmlToMarkdownBuilder::new()
    .add_handler(
        vec!["iframe"],
        |_handlers: &dyn Handlers, element: htmd::Element| {
            let src = element
                .attrs
                .iter()
                .find(|a| a.name.local.as_ref() == "src")
                .map(|a| a.value.to_string())?;
            let url = if src.contains("youtube.com/embed/") {
                src.replace("/embed/", "/watch?v=")
            } else {
                src
            };
            Some(HandlerResult::from(format!("\n![]({url})")))
        },
    )
    .add_handler(vec!["sup"], handle_sup_element)
    .add_handler(vec!["span", "div"], handle_span_div_element)
    .add_handler(vec!["a"], handle_anchor_element)
    .build()
    .convert(html)
    .ok()
    .map(|s| fix_bang_image_collision(s.as_str()))
    .map(|s| unescape_latex_delimiters(&s))
    .map(|s| clean_bare_bullets(&s))
    .map(|s| collapse_newlines(&s))
}
```

--------------------------------

### Any Trait Implementation

Source: https://docs.rs/decruft/0.1.4/decruft/struct.DebugInfo.html?search=

Provides a way to get the TypeId of a value, useful for dynamic type checking.

```rust
fn type_id(&self) -> TypeId
```

--------------------------------

### Build Post Content HTML

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/comments.rs.html?search=u32+-%3E+bool

Constructs the main HTML for a post, optionally including a comments section. Use this to generate the overall page structure.

```Rust
pub fn build_content_html(site: &str, post_content: &str, comments: &str) -> String {
    let post_section = format!(
        "<div class=\"{site}\" post"><div class=\"post-content\">{post_content}</div></div>"
    );
    if comments.is_empty() {
        return post_section;
    }
    format!("{post_section}<hr><div class=\"{site}\" comments"><h2>Comments</h2>{comments}</div>")
}
```

--------------------------------

### Fetch Stack Exchange Question

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/stackoverflow.rs.html

Constructs a URL to fetch a specific question from the Stack Exchange API and returns the JSON response.

```rust
fn fetch_se_question(id: &str, site: &str) -> Option<serde_json::Value> {
    let url = format!(
        "{SE_API_BASE}/questions/{id}\
         ?site={site}&filter=withbody&order=desc&sort=activity"
    );
    fetch_se_json(&url)
}
```

--------------------------------

### Check for class with prefix

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/substack.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Helper function to determine if an element has a class that starts with a given prefix.

```rust
fn has_class_with_prefix(el: &scraper::node::Element, prefix: &str) -> bool {
    el.attr("class")
        .is_some_and(|c| c.split_whitespace().any(|cls| cls.starts_with(prefix)))
}
```

--------------------------------

### Build Comment Title

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/hackernews.rs.html?search=

Constructs a title string for a comment, including the author and a preview of the comment's content. The preview is truncated to 50 characters.

```Rust
fn build_comment_title(comment: &CommentData) -> String {
    let text = dom::strip_html_tags(&comment.content);
    let trimmed = text.trim();
    let preview = match trimmed.char_indices().nth(50) {
        Some((i, _)) => format!("{}\
...", &trimmed[..i]),
        None => trimmed.to_string(),
    };
    format!("Comment by {}: {preview}", comment.author)
}
```

--------------------------------

### Test API Fetch on Live Page

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/c2wiki.rs.html

Checks the `try_api_fetch` function by attempting to retrieve content for a known wiki page. It asserts that the content is substantial and that the title and site information are correctly extracted.

```Rust
#[test]
fn api_fetch_on_live_page() {
    let result = try_api_fetch("ExtremeProgramming");
    if let Some(r) = result {
        assert!(r.content.len() > 100, "should have substantial content");
        assert_eq!(r.title, Some("Extreme Programming".to_string()));
        assert_eq!(r.site, Some("C2 Wiki".to_string()));
    }
    // Don't fail if network is unavailable
}
```

--------------------------------

### Load Fixture for Tests

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/reddit.rs.html

Loads HTML content from a test fixture file. Panics if the file is not found.

```Rust
fn load_fixture(name: &str) -> String {
        let path = format!("{}/tests/fixtures/{name}", env!("CARGO_MANIFEST_DIR"));
        std::fs::read_to_string(&path)
            .unwrap_or_else(|e| panic!("fixture not found at {path}: {e}"))
    }
```

--------------------------------

### Find Element with Class Prefix

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/substack.rs.html?search=

Checks if an element with a class name starting with the given prefix exists.

```Rust
fn has_class_prefix(html: &Html, prefix: &str) -> bool {
    find_element_with_class_prefix(html, prefix).is_some()
}
```

--------------------------------

### Extract Language from HTML Attribute

Source: https://docs.rs/decruft/0.1.4/src/decruft/metadata.rs.html?search=

This example shows how to extract the page language from the 'lang' attribute of the '<html>' tag.

```rust
let doc = Html::parse_document(r"<html lang=\"en-US\"><body></body></html>");
        let m = extract_metadata(&doc, None, None);
        assert_eq!(m.language.as_deref(), Some("en-US"));
```

--------------------------------

### Build Comment Title

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/hackernews.rs.html?search=std%3A%3Avec

Constructs a title for a comment, including the author and a preview of the comment's content.

```Rust
fn build_comment_title(comment: &CommentData) -> String {
    let text = dom::strip_html_tags(&comment.content);
    let trimmed = text.trim();
    let preview = match trimmed.char_indices().nth(50) {
        Some((i, _)) => format!("{}...", &trimmed[..i]),
        None => trimmed.to_string(),
    };
    format!("Comment by {}: {preview}", comment.author)
}
```

--------------------------------

### Get Site Name from ConversationSite Enum

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/conversations.rs.html

Returns the static string name for a given ConversationSite enum variant.

```rust
fn site_name(site: &ConversationSite) -> &'static str {
    match site {
        ConversationSite::ChatGpt => "ChatGPT",
        ConversationSite::Claude => "Claude",
        ConversationSite::Gemini => "Gemini",
        ConversationSite::Grok => "Grok",
    }
}
```

--------------------------------

### Format Base URL

Source: https://docs.rs/decruft/0.1.4/src/decruft/metadata.rs.html?search=

Constructs a base URL string including scheme, host, and port if non-default. Requires a valid URL string.

```rust
/// Return scheme + host (+ port if non-default).
fn base_url(raw: &str) -> Option<String> {
    let parsed = url::Url::parse(raw).ok()?;
    let host = parsed.host_str()?;
    let scheme = parsed.scheme();
    match parsed.port() {
        Some(port) => Some(format!("{scheme}://{host}:{port}")),
        None => Some(format!("{scheme}://{host}")),
    }
}
```

--------------------------------

### Skip Line Comments in Rust

Source: https://docs.rs/decruft/0.1.4/src/decruft/schema_org.rs.html?search=

This snippet demonstrates how to skip single-line comments starting with '//' in a character stream.

```Rust
if ch == '/' {
            chars.next();
            if let Some(&next) = chars.peek() {
                if next == '/' {
                    chars.next();
                    // Skip until newline
                    while let Some(&c) = chars.peek() {
                        chars.next();
                        if c == '\n' {
                            break;
                        }
                    }
                    continue;
                }
                // ... other comment types
            }
            result.push('/');
            continue;
        }
```

--------------------------------

### Markdown to HTML Sanitizes JavaScript Links

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html

Tests that links starting with 'javascript:' are sanitized and not converted to clickable links.

```rust
#[test]
fn markdown_to_html_sanitizes_javascript_links() {
    let md = "[click](javascript:alert(1))";
    let html = markdown_to_html(md);
    assert!(!html.contains("javascript:"));
}
```

--------------------------------

### Build Content HTML Function

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/comments.rs.html

Constructs the complete HTML for a post, optionally including a comments section. Handles cases where comments are empty.

```rust
pub fn build_content_html(site: &str, post_content: &str, comments: &str) -> String {
    let post_section = format!(
        "<div class=\"{site}\" post"><div class=\"post-content\">{post_content}</div></div>"
    );
    if comments.is_empty() {
        return post_section;
    }
    format!("{post_section}<hr><div class=\"{site}\" comments><h2>Comments</h2>{comments}</div>")
}
```

--------------------------------

### Try API Fetch

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html?search=u32+-%3E+bool

Attempts to fetch and parse story data from the Lobste.rs API.

```APIDOC
## try_api_fetch

### Description
Fetches story data from the Lobste.rs JSON API and builds an `ExtractorResult`.

### Function Signature
`fn try_api_fetch(url: Option<&str>, include_replies: bool) -> Option<ExtractorResult>`

### Parameters
* `url` (Option<&str>) - An optional string slice representing the Lobste.rs URL.
* `include_replies` (bool) - A boolean indicating whether to include comments in the fetched data.

### Returns
* `Option<ExtractorResult>` - An optional `ExtractorResult` containing the parsed story data, or `None` if fetching or parsing fails.
```

--------------------------------

### Sanitize JavaScript Links

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html?search=

Verifies that links starting with 'javascript:' are removed to prevent execution of arbitrary JavaScript code.

```rust
    #[test]
    fn markdown_to_html_sanitizes_javascript_links() {
        let md = "[click](javascript:alert(1))";
        let html = markdown_to_html(md);
        assert!(!html.contains("javascript:"));
    }
```

--------------------------------

### Fetch and Build from API

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html?search=std%3A%3Avec

Fetches data from the Lobste.rs JSON API and builds an `ExtractorResult`.

```APIDOC
## try_api_fetch

### Description
Attempts to fetch data from the Lobste.rs JSON API and parse it into an `ExtractorResult`. It takes an optional URL and a flag to include replies.

### Function Signature
`fn try_api_fetch(url: Option<&str>, include_replies: bool) -> Option<ExtractorResult>`

### Parameters
* `url` (Option<&str>) - An optional string slice representing the Lobste.rs story URL.
* `include_replies` (bool) - A boolean flag indicating whether to include comments in the result.

### Returns
* `Option<ExtractorResult>` - An optional `ExtractorResult` containing the extracted content, title, author, site, and publication date if the fetch and parsing are successful, otherwise `None`.
```

--------------------------------

### Get Dublin Core Meta Content

Source: https://docs.rs/decruft/0.1.4/src/decruft/metadata.rs.html?search=

Retrieves content for Dublin Core fields by checking common naming variants.

```rust
/// Query Dublin Core meta tags. Checks all common naming variants:
/// `DC.{field}`, `dc.{field}`, `dc:{field}`, `dcterm:{field}`,
/// `DCTERMS.{field}`, `dcterms.{field}`.
fn get_dc_content(html: &Html, field: &str) -> Option<String> {
    let variants = [
        format!("DC.{field}"),
        format!("dc.{field}"),
        format!("dc:{field}"),
        format!("dcterm:{field}"),
        format!("DCTERMS.{field}"),
        format!("dcterms.{field}"),
    ];
    for name in &variants {
        if let Some(v) = get_meta_content(html, "name", name) {
            return Some(v);
        }
    }
    None
}

```

--------------------------------

### Get Text Content from JSON

Source: https://docs.rs/decruft/0.1.4/src/decruft/schema_org.rs.html

Extracts the primary text content from a JSON object, typically from a field like 'articleBody'.

```rust
fn test_get_text() {
        let data: Value = 
            serde_json::from_str(r#"{"@type": "Article", "articleBody": "Hello world"}"#)
                .ok()
                .unwrap_or_default();
        assert_eq!(get_text(&data), Some("Hello world".to_string()));
    }
```

--------------------------------

### Skip Line and Block Comments

Source: https://docs.rs/decruft/0.1.4/src/decruft/schema_org.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Skips over single-line comments starting with '//' and multi-line comments enclosed in '/* ... */'.

```rust
        // Line comment
        if ch == '/' {
            chars.next();
            if let Some(&next) = chars.peek() {
                if next == '/' {
                    chars.next();
                    // Skip until newline
                    while let Some(&c) = chars.peek() {
                        chars.next();
                        if c == '\n' {
                            break;
                        }
                    }
                    continue;
                }
                if next == '*' {
                    chars.next();
                    // Skip until */
                    let mut prev = ' ';
                    while let Some(&c) = chars.peek() {
                        chars.next();
                        if prev == '*' && c == '/' {
                            break;
                        }
                        prev = c;
                    }
                    continue;
                }
            }
            result.push('/');
            continue;
        }
```

--------------------------------

### Fetch Live GitHub Issue Data with Labels

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html?search=

Fetches live data for a GitHub issue, verifying that the content is parseable and non-empty. Does not fail if the network is unavailable.

```rust
#[test]
    fn api_fetch_live_issue_with_labels() {
        // Use a well-known issue that has labels
        let url = "https://github.com/rust-lang/rust/issues/1";
        let result = try_api_fetch(Some(url), false);
        if let Some(r) = result {
            assert!(r.title.is_some());
            // Issue #1 doesn't necessarily have labels, but the
            // content should be parseable and non-empty
            assert!(!r.content.is_empty());
        }
        // Don't fail if network is unavailable
    }
```

--------------------------------

### Get Simple JSON Property

Source: https://docs.rs/decruft/0.1.4/src/decruft/schema_org.rs.html

Retrieves a simple property value from a JSON object, supporting dot notation for nested properties.

```rust
fn test_get_property_simple() {
        let data: Value = 
            serde_json::from_str(r#"{"author": {"name": "Alice"}, "datePublished": "2024"}"#)
                .ok()
                .unwrap_or_default();
        assert_eq!(
            get_property(&data, "author.name"),
            Some("Alice".to_string())
        );
        assert_eq!(
            get_property(&data, "datePublished"),
            Some("2024".to_string())
        );
    }
```

--------------------------------

### Generic Blanket Implementations

Source: https://docs.rs/decruft/0.1.4/decruft/struct.DecruftOptions.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Demonstrates various blanket implementations available for generic types, including Any, Borrow, CloneToUninit, From, Into, ToOwned, TryFrom, and TryInto.

```rust
fn type_id(&self) -> TypeId
```

```rust
fn borrow(&self) -> &T
```

```rust
fn borrow_mut(&mut self) -> &mut T
```

```rust
unsafe fn clone_to_uninit(&self, dest: *mut u8)
```

```rust
fn from(t: T) -> T
```

```rust
fn into(self) -> U
```

```rust
type Owned = T
```

```rust
fn to_owned(&self) -> T
```

```rust
fn clone_into(&self, target: &mut T)
```

```rust
type Error = Infallible
```

```rust
fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
```

```rust
type Error = <U as TryFrom<T>>::Error
```

```rust
fn try_from(value: U) -> Result<T, <U as TryFrom<T>>::Error>
```

--------------------------------

### Find element with class prefix

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/substack.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Searches the HTML tree for the first element whose class attribute starts with the specified prefix.

```rust
fn find_element_with_class_prefix(html: &Html, prefix: &str) -> Option<ego_tree::NodeId> {
    for node_ref in html.tree.nodes() {
        if let scraper::Node::Element(el) = node_ref.value()
            && has_class_with_prefix(el, prefix)
        {
            return Some(node_ref.id());
        }
    }
    None
}
```

--------------------------------

### Markdown to HTML Sanitizes Data URI Links

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html

Tests that links starting with 'data:' are sanitized to prevent execution of embedded scripts.

```rust
#[test]
fn markdown_to_html_sanitizes_data_uri_links() {
    let md = "[click](data:text/html,<script>alert(1)</script>)";
    let html = markdown_to_html(md);
    assert!(!html.contains("data:text/html"));
}
```

--------------------------------

### Serialize Implementation for DebugInfo

Source: https://docs.rs/decruft/0.1.4/decruft/struct.DebugInfo.html?search=

Enables DebugInfo instances to be serialized into a Serde serializer.

```rust
fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>
where __S: Serializer
```

--------------------------------

### From and Into Implementations

Source: https://docs.rs/decruft/0.1.4/decruft/struct.MetaTag.html?search=std%3A%3Avec

Enables conversion to and from other types that implement `From`.

```rust
fn from(t: T) -> T
```

```rust
fn into(self) -> U
where U: From<T>,
```

--------------------------------

### URL Encoding Basics

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/twitter.rs.html?search=std%3A%3Avec

Tests the `urlencoding` function for basic URL string encoding.

```rust
#[test]
fn urlencoding_basics() {
    assert_eq!(
        urlencoding("https://x.com/user/status/123"),
        "https%3A%2F%2Fx.com%2Fuser%2Fstatus%2F123"
    );
    assert_eq!(urlencoding("hello"), "hello");
}
```

--------------------------------

### Parse Short ID from Lobste.rs URL

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html

Extracts the short ID from a Lobste.rs story URL. For example, `https://lobste.rs/s/abc123/story_title` yields `abc123`.

```rust
fn parse_short_id(url: &str) -> Option<&str> {
    let after_s = url.split("lobste.rs/s/").nth(1)?;
    let id = after_s.split('/').next()?;
    if id.is_empty() {
        return None;
    }
    Some(id)
}
```

--------------------------------

### Fetch JSON from URL

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/stackoverflow.rs.html

Fetches data from a given URL using an HTTP GET request and attempts to parse the response body as JSON.

```rust
fn fetch_se_json(url: &str) -> Option<serde_json::Value> {
    let body = crate::http::get(url)?;
    serde_json::from_str(&body).ok()
}
```

--------------------------------

### Fetch Live GitHub PR with Review Comments

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html?search=u32+-%3E+bool

Fetches data for a live GitHub pull request to verify that review comments, including replies, are correctly included in the content. Does not fail if the network is unavailable.

```Rust
let url = "https://github.com/rust-lang/rust/pull/2";
let result = try_api_fetch(Some(url), true);
if let Some(r) = result {
    assert!(r.title.is_some());
    assert!(!r.content.is_empty());
}
```

--------------------------------

### Get Attribute Value from Element

Source: https://docs.rs/decruft/0.1.4/src/decruft/dom.rs.html

Retrieves the value of a specified attribute from an element node. Returns `None` if the node is not found, is not an element, or does not have the attribute.

```rust
/// Get an attribute value from an element node.
pub fn get_attr(html: &Html, node_id: NodeId, attr: &str) -> Option<String> {
```

--------------------------------

### Test Live API Fetch

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html

Tests the `try_api_fetch` function with a live URL, focusing on ensuring it does not crash. The result is dropped without assertions.

```rust
#[test]
fn api_fetch_live() {
    let url = "https://lobste.rs/s/abc123";
    let result = try_api_fetch(Some(url), false);
    // Don't assert success — just verify it doesn't crash
    drop(result);
}
```

--------------------------------

### Remove Boundary Time

Source: https://docs.rs/decruft/0.1.4/src/decruft/patterns.rs.html

Removes parent elements (p or div) containing time elements if they are near the start or end of content and have few words.

```rust
fn remove_boundary_time(
    html: &mut Html,
    time_id: NodeId,
    text: &str,
    removals: &mut Vec<Removal>,
    debug: bool,
) {
    // Check if it's near the start or end of content
    if let Some(parent_id) = dom::parent_element(html, time_id) {
        let parent_tag = dom::tag_name(html, parent_id);
        if parent_tag.as_deref() == Some("p") || parent_tag.as_deref() == Some("div") {
            let parent_text = dom::text_content(html, parent_id);
            let parent_words = dom::count_words(&parent_text);
            if parent_words <= 8 && DATE_RE.is_match(&text) {
                record_removal(removals, debug, "boundary time", &text);
                to_remove.push(parent_id);
            }
        }
    }
}

for id in to_remove {
    dom::remove_node(html, id);
}
}
```

--------------------------------

### Run Standardize Function

Source: https://docs.rs/decruft/0.1.4/src/decruft/footnotes.rs.html

Helper function to parse HTML input and apply the `standardize_footnotes` transformation. It selects the main content area (article or body) before standardization.

```Rust
fn run_standardize(input: &str) -> Html {
    let mut html = Html::parse_document(input);
    let article = dom::select_ids(&html, "article");
    let main_id = if article.is_empty() {
        dom::select_ids(&html, "body")[0]
    } else {
        article[0]
    };
    standardize_footnotes(&mut html, main_id);
    html
}
```

--------------------------------

### Fetch Live GitHub Issue with Labels

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html?search=u32+-%3E+bool

Fetches data for a live GitHub issue to ensure content is parseable and non-empty, even if specific labels are not guaranteed. Does not fail if the network is unavailable.

```Rust
let url = "https://github.com/rust-lang/rust/issues/1";
let result = try_api_fetch(Some(url), false);
if let Some(r) = result {
    assert!(r.title.is_some());
    // Issue #1 doesn't necessarily have labels, but the
    // content should be parseable and non-empty
    assert!(!r.content.is_empty());
}
```

--------------------------------

### Check for Dangerous URL Schemes

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html?search=

Determines if a URL starts with potentially dangerous schemes like 'javascript:', 'vbscript:', or 'data:'. The check is case-insensitive.

```Rust
/// Check if a link destination is dangerous (case-insensitive).
fn is_dangerous_link(url: &str) -> bool {
    let lower = url.trim().to_ascii_lowercase();
    lower.starts_with("javascript:") || lower.starts_with("vbscript:") || lower.starts_with("data:")
}
```

--------------------------------

### Check for Safe URL Scheme

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/bbcode.rs.html?search=

Validates if a given URL has a safe scheme (http, https, or starts with '/') for use in HTML attributes.

```rust
/// Check if a URL has a safe scheme for use in href attributes.
fn is_safe_url(url: &str) -> bool {
    let trimmed = url.trim().to_ascii_lowercase();
    trimmed.starts_with("http://") || trimmed.starts_with("https://") || trimmed.starts_with('/')
}
```

--------------------------------

### Try Site-Specific Extractors

Source: https://docs.rs/decruft/0.1.4/src/decruft/decruft.rs.html

Attempts to extract content using site-specific extractors like GitHub or Reddit. Returns `None` if no content is extracted or if the word count is zero.

```rust
fn try_site_extractors(
    html: &Html,
    start: &Instant,
    options: &DecruftOptions,
    meta: &mut crate::types::Metadata,
    schema_data: Option<&serde_json::Value>,
    meta_tags: &[crate::types::MetaTag],
) -> Option<DecruftResult> {
    let (extracted, extractor_name) =
        extractors::try_extract(html, options.url.as_deref(), options.include_replies)?;
    apply_extractor_metadata(&extracted, meta);
    let word_count = dom::count_words_html(&extracted.content);
    if word_count == 0 {
        return None;
    }
    let mut result = build_extractor_result(
        &extracted.content,
        "site-extractor",
        start,
        options,
        meta,
        schema_data,
        meta_tags,
        word_count,
    );
    result.extractor_type = Some(extractor_name.to_string());
    Some(result)
}
```

--------------------------------

### Build Lobste.rs API URL

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html

Constructs the JSON API URL for a Lobste.rs story using its short ID. Returns `None` if the short ID cannot be parsed from the input URL.

```rust
fn api_url(url: &str) -> Option<String> {
    let id = parse_short_id(url)?;
    Some(format!("https://lobste.rs/s/{id}.json"))
}
```

--------------------------------

### Get JSON Property from Array Element

Source: https://docs.rs/decruft/0.1.4/src/decruft/schema_org.rs.html

Retrieves a property value from an element within a JSON array, using bracket notation for array indexing.

```rust
fn test_get_property_array() {
        let data: Value = serde_json::from_str(r#"{"author": [{"name": "A"}, {"name": "B"}]}"#)
            .ok()
            .unwrap_or_default();
        assert_eq!(get_property(&data, "author.[].name"), Some("A".to_string()));
    }
```

--------------------------------

### Fetch Live GitHub Issue Data with Labels

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html?search=std%3A%3Avec

Fetches live data for a GitHub issue, verifying content is parseable and non-empty. Useful for checking issue content integrity.

```rust
// Use a well-known issue that has labels
let url = "https://github.com/rust-lang/rust/issues/1";
let result = try_api_fetch(Some(url), false);
if let Some(r) = result {
    assert!(r.title.is_some());
    // Issue #1 doesn't necessarily have labels, but the
    // content should be parseable and non-empty
    assert!(!r.content.is_empty());
}
// Don't fail if network is unavailable
```

--------------------------------

### Get Tag Name of an Element Node

Source: https://docs.rs/decruft/0.1.4/src/decruft/dom.rs.html?search=

Extracts the local name of an element node. Returns None if the node ID is invalid or the node is not an element.

```rust
let node_ref = html.tree.get(node_id)?;
    if let Node::Element(el) = node_ref.value() {
        Some(el.name.local.as_ref().to_string())
    } else {
        None
    }
```

--------------------------------

### Get an Attribute Value from a DOM Node

Source: https://docs.rs/decruft/0.1.4/src/decruft/dom.rs.html?search=

Retrieves the value of a specified attribute from an element node. Returns `None` if the node does not exist or is not an element with the attribute.

```rust
/// Get an attribute value from an element node.
pub fn get_attr(html: &Html, node_id: NodeId, attr: &str) -> Option<String> {
    let node_ref = html.tree.get(node_id)?;

```

--------------------------------

### Fetch Live GitHub Pull Request with Review Comments

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/github.rs.html?search=

Fetches live data for a GitHub pull request, including replies, to verify that review comments are correctly included in the content. Does not fail if the network is unavailable.

```rust
#[test]
    fn api_fetch_live_pr_includes_review_comments() {
        // Fetch a PR with replies to verify review comments are included
        let url = "https://github.com/rust-lang/rust/pull/2";
        let result = try_api_fetch(Some(url), true);
        if let Some(r) = result {
            assert!(r.title.is_some());
            assert!(!r.content.is_empty());
        }
        // Don't fail if network is unavailable
    }
```

--------------------------------

### Get Text Content of a Node

Source: https://docs.rs/decruft/0.1.4/src/decruft/dom.rs.html

Recursively collects and returns the text content of a given node and all its descendants. Requires an initialized Html object and a NodeId.

```rust
use std::fmt::Write;

use ego_tree::NodeId;
use scraper::{Html, Node, Selector};

/// Get the text content of a node and its descendants.
#[must_use]
pub fn text_content(html: &Html, node_id: NodeId) -> String {
    let mut text = String::new();
    collect_text(html, node_id, &mut text);
    text
}

fn collect_text(html: &Html, node_id: NodeId, buf: &mut String) {
    let node_ref = html.tree.get(node_id);
    let Some(node_ref) = node_ref else { return };
    match node_ref.value() {
        Node::Text(t) => buf.push_str(t),
        Node::Element(_) => {
            for child in node_ref.children() {
                collect_text(html, child.id(), buf);
            }
        }
        _ => {}
    }
}
```

--------------------------------

### Fetch URL with Custom Headers and API Defaults

Source: https://docs.rs/decruft/0.1.4/src/decruft/http.rs.html

Fetches content from a URL, allowing custom headers to be added to the request. It uses the default API timeout and User-Agent. Returns `None` on request failures.

```Rust
pub(crate) fn get_with_headers(url: &str, headers: &[(&str, &str)]) -> Option<String> {
    let agent = build_agent(API_TIMEOUT);
    let mut request = agent.get(url).header("User-Agent", API_USER_AGENT);
    for &(name, value) in headers {
        request = request.header(name, value);
    }
    let response = request.call().ok()?;
    if response.status() != 200 {
        return None;
    }
    response.into_body().read_to_string().ok()
}
```

--------------------------------

### Find Element with Class Prefix

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/substack.rs.html?search=u32+-%3E+bool

Helper function to find an element that has a class starting with the given prefix. Returns the element's NodeId if found.

```rust
fn find_element_with_class_prefix(html: &Html, prefix: &str) -> Option<ego_tree::NodeId> {
    for node_ref in html.tree.nodes() {
        if let scraper::Node::Element(el) = node_ref.value() {
            if has_class_with_prefix(el, prefix) {
                return Some(node_ref.id());
            }
        }
    }
    None
}
```

--------------------------------

### Build API URL

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html?search=

Constructs the JSON API URL for a given Lobste.rs story URL.

```APIDOC
## `api_url` function

### Description
Builds the JSON API URL from a Lobste.rs story page URL.

### Parameters
- `url` (string): The Lobste.rs story URL.

### Returns
- `Option<String>`: The constructed API URL, or `None` if the short ID cannot be parsed.
```

--------------------------------

### Build API URL

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/lobsters.rs.html

Constructs the JSON API URL for a given Lobste.rs story URL.

```APIDOC
## fn api_url(url: &str) -> Option<String>

### Description
Builds the JSON API URL for a Lobste.rs story using its URL. It first parses the short ID from the provided URL and then formats it into the API endpoint.

### Parameters
* `url` (string) - The Lobste.rs story URL.

### Returns
* `Option<String>` - An optional String containing the API URL if the short ID can be parsed, otherwise `None`.
```

--------------------------------

### Check for Class Prefix Presence

Source: https://docs.rs/decruft/0.1.4/src/decruft/extractors/substack.rs.html?search=std%3A%3Avec

Checks if any element on the page has a class that starts with the given prefix. This is a utility function for identifying specific element types.

```rust
/// Check if any element on the page has a class with the given prefix.

```

--------------------------------

### Extract Author from Meta Tag

Source: https://docs.rs/decruft/0.1.4/src/decruft/metadata.rs.html?search=

This example shows how to extract the author's name from a 'meta name="author"' tag in the HTML head.

```rust
let doc = Html::parse_document(
            r#"<html><head>
            <meta name=\"author\" content=\"Jane Doe\">
            </head><body></body></html>"#,
        );
        let m = extract_metadata(&doc, None, None);
        assert_eq!(m.author.as_deref(), Some("Jane Doe"));
```
Tokens:
Raw