url

https://docs.rs/url
Admin
url is a Rust library that implements the WHATWG URL Standard, providing URL parsing, manipulation,...
Tokens:134,119
Snippets:1,102
Trust Score:9.1
Update:1 month ago
Show doc for...
Tokens:
Raw
### Example output for ToQueryString

Source: https://docs.rs/url_query_string/latest/url_query_string

The expected output when running the TestStruct example.

```text
Query String: page=1&pageSize=20&id=test_id&userId=user_123
Query String (with Result): page=1&pageSize=20&id=test_id&userId=user_123
```

--------------------------------

### Example URL

Source: https://docs.rs/url_pest_parser/latest/url_pest_parser

An example URL that conforms to the url rule pattern.

```text
https://www.example.com:8080/path/to/resource?key=value#section1
```

--------------------------------

### Basic usage example

Source: https://docs.rs/url-scraper/latest/url_scraper

Shows how to create a UrlScraper instance and iterate over the URLs found on a webpage. The example prints each text and URL pair.

```rust
extern crate url_scraper;
use url_scraper::UrlScraper;

let scraper = UrlScraper::new("http://phoronix.com/").unwrap();
for (text, url) in scraper.into_iter() {
    println!("{}: {}", text, url);
}
```

--------------------------------

### Valid URL example using url! macro

Source: https://docs.rs/url-macro/latest/url_macro

This example shows a valid URL that compiles correctly with the url! macro. Use this to ensure your URL is valid at compile time.

```rust
// This compiles correctly
let valid = url!("https://www.rust-lang.org/");
```

--------------------------------

### Using with reqwest: Resolve to Verified IP

Source: https://docs.rs/url_jail/latest/url_jail

Shows how to use the Validated struct with reqwest's resolver override to avoid a second DNS lookup. The example builds a Client with the resolved IP and then sends a GET request.

```rust
use url_jail::{validate, Policy};
use reqwest::Client;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
    let v = validate("https://example.com/api", Policy::PublicOnly).await?;

    let client = Client::builder()
        .resolve(&v.host, v.to_socket_addr())
        .build()?;

    let response = client.get(&v.url).send().await?;
    Ok(())
}
```

--------------------------------

### Option::zip example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Combines two Options into a tuple if both are Some, otherwise returns None. The example shows zipping with a Some and a None.

```rust
let x = Some(1);
let y = Some("hi");
let z = None::<u8>;

assert_eq!(x.zip(y), Some((1, "hi")));
assert_eq!(x.zip(z), None);
```

--------------------------------

### url2! macro usage example

Source: https://docs.rs/url2/latest/url2

Shows how to use the url2! macro to create a URL, set and remove query parameters, and assert the resulting string. The example uses the query_unique() method to set and remove pairs, and checks the final URL string.

```rust
#[macro_use]
extern crate url2;

fn main() {
    let mut url = url2!("https://{}/", "example.com");
    url.query_unique()
        .set_pair("hello", "world")
        .set_pair("foo", "bar");

    assert!(url.query_unique_contains_key("hello"));
    assert_eq!("bar", url.query_unique_get("foo").unwrap());

    url.query_unique().remove("foo");

    assert_eq!(
        "https://example.com/?hello=world",
        url.as_str(),
    )
}
```

--------------------------------

### Quick Start with PreviewService

Source: https://docs.rs/url-preview/latest/url_preview

Basic usage of the PreviewService to generate a URL preview.

```rust
use url_preview::{PreviewService, PreviewError};

#[tokio::main]
async fn main() -> Result<(), PreviewError> {
    let service = PreviewService::new();
    let preview = service.generate_preview("https://www.rust-lang.org").await?;

    println!("Title: {:?}", preview.title);
    println!("Description: {:?}", preview.description);
    Ok(())
}
```

--------------------------------

### Valid URL macro examples

Source: https://docs.rs/url-static/latest/url_static

Shows the `url!` macro accepting various valid URL strings, including HTTPS, HTTP, mailto, and a URL from an environment variable. These examples compile and run successfully.

```rust
use url_static::url;

// ✅ These will for sure work:
let api = url!("https://api.example.com");
assert_eq!(api, Url::parse("https://api.example.com").unwrap());

let test = url!("http://localhost/");
assert_eq!(test, Url::parse("http://localhost/").unwrap());

let mailto = url!("mailto:also_works_for_mailto@example.com");
assert_eq!(mailto, Url::parse("mailto:also_works_for_mailto@example.com").unwrap());

// ✅ It even works with other macros:
let repo = url!(env!("CARGO_PKG_REPOSITORY"));
assert_eq!(repo, Url::parse(env!("CARGO_PKG_REPOSITORY")).unwrap());
```

--------------------------------

### Option::from for T example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Moves a value into a new Some. The example shows converting a u8 value into an Option<u8>.

```rust
let o: Option<u8> = Option::from(67);

assert_eq!(Some(67), o);
```

--------------------------------

### QueryParams::from and stringify usage example

Source: https://docs.rs/urlqstring/latest/urlqstring

Shows how to construct a QueryParams from a vector of key-value pairs and stringify it to a query string. The example asserts that the output matches the expected encoded string, including a trailing ampersand.

```rust
use urlqstring::QueryParams;
fn main() {
    let enc1 = QueryParams::from(vec![
        ("params", "度"),
        ("enc", "=-)(&#+~·@…%^$:*-_."),
    ]).stringify();
    let res1: String= String::from("params=%E5%BA%A6&enc=%3D-)(%26%23%2B~%C2%B7%40%E2%80%A6%25%5E%24%3A*-_.&");
    assert_eq!(enc1, res1);
}
```

--------------------------------

### expect recommended message style

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Example of using expect to describe why a value is expected to be Some.

```rust
let item = slice.get(0)
    .expect("slice should not be empty");
```

--------------------------------

### Example output for snake_case configuration

Source: https://docs.rs/url_query_string/latest/url_query_string

The resulting query string format when using snake_case.

```text
user_name=test_user&access_token=abcd1234
```

--------------------------------

### Relative URL reference example

Source: https://docs.rs/url

An example of a relative URL reference often found in HTML.

```html
<link rel="stylesheet" href="../main.css">
```

--------------------------------

### Option::from for &Option<T> example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Converts a reference to an Option into an Option of a reference, preserving the original. The example converts an Option<String> to an Option<usize> using map and from, and shows the original can still be printed.

```rust
let s: Option<String> = Some(String::from("Hello, Rustaceans!"));
let o: Option<usize> = Option::from(&s).map(|ss: &String| ss.len());

println!("Can still print s: {s:?}");

assert_eq!(o, Some(18));
```

--------------------------------

### Option::zip_with example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Combines two Options using a function if both are Some, otherwise returns None. This is a nightly-only experimental API requiring the option_zip feature. The example defines a Point struct and uses zip_with with Point::new.

```rust
#![feature(option_zip)]

#[derive(Debug, PartialEq)]
struct Point {
    x: f64,
    y: f64,
}

impl Point {
    fn new(x: f64, y: f64) -> Self {
        Self { x, y }
    }
}

let x = Some(17.5);
let y = Some(42.7);

assert_eq!(x.zip_with(y, Point::new), Some(Point { x: 17.5, y: 42.7 }));
assert_eq!(x.zip_with(None, Point::new), None);
```

--------------------------------

### Option::default example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Returns None for the default value of Option. The example shows that Option::default() returns None.

```rust
let opt: Option<u32> = Option::default();
assert!(opt.is_none());
```

--------------------------------

### Option::from_iter example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Collects an iterator of Options into an Option of a container. If any element is None, the result is None. The example shows collecting checked_add results and checked_sub results.

```rust
let items = vec![0_u16, 1, 2];

let res: Option<Vec<u16>> = items
    .iter() n    .map(|x| x.checked_add(1))
    .collect(); n
assert_eq!(res, Some(vec![1, 2, 3]));
```

```rust
let items = vec![2_u16, 1, 0];
```

--------------------------------

### Option::cloned Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Demonstrates cloning the contents of an Option<&T>.

```rust
let x = 12;
let opt_x = Some(&x);
assert_eq!(opt_x, Some(&12));
let cloned = opt_x.cloned();
assert_eq!(cloned, Some(12));
```

--------------------------------

### Option::copied Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Demonstrates copying the contents of an Option<&T>.

```rust
let x = 12;
let opt_x = Some(&x);
assert_eq!(opt_x, Some(&12));
let copied = opt_x.copied();
assert_eq!(copied, Some(12));
```

--------------------------------

### Manual URL Prefix Formatting

Source: https://docs.rs/url-prefix/latest/url_prefix

An example of the verbose manual string manipulation required to format a URL prefix without using the crate.

```rust
let mut url_prefix = String::new();

if is_https {
    url_prefix.push_str("https://");
} else {
    url_prefix.push_str("http://");
}

url_prefix.push_str(domain);

if is_https && port != 443 || !is_https && port != 80 {
    url_prefix.push_str(":");
    url_prefix.push_str(&port.to_string());
}
```

--------------------------------

### Path Start Parsing

Source: https://docs.rs/url/latest/src/url/parser.rs.html

Initializes the path state for a URL, handling special scheme requirements and identifying query or fragment delimiters.

```rust
    pub fn parse_path_start<'i>(
        &mut self,
        scheme_type: SchemeType,
        has_host: &mut bool,
        input: Input<'i>,
    ) -> Input<'i> {
        let path_start = self.serialization.len();
        let (maybe_c, remaining) = input.split_first();
        // If url is special, then:
        if scheme_type.is_special() {
            if maybe_c == Some('\') {
                // If c is U+005C (\), validation error.
                self.log_violation(SyntaxViolation::Backslash);
            }
            // A special URL always has a non-empty path.
            if !self.serialization.ends_with('/') {
                self.serialization.push('/');
                // We have already made sure the forward slash is present.
                if maybe_c == Some('/') || maybe_c == Some('\') {
                    return self.parse_path(scheme_type, has_host, path_start, remaining);
                }
            }
            return self.parse_path(scheme_type, has_host, path_start, input);
        } else if maybe_c == Some('?') || maybe_c == Some('#') {
            // Otherwise, if state override is not given and c is U+003F (?),
            // set url’s query to the empty string and state to query state.
            // Otherwise, if state override is not given and c is U+0023 (#),
            // set url’s fragment to the empty string and state to fragment state.
            // The query and path states will be handled by the caller.
            return input;
        }

        if maybe_c.is_some() && maybe_c != Some('/') {
            self.serialization.push('/');
        }
        // Otherwise, if c is not the EOF code point:
        self.parse_path(scheme_type, has_host, path_start, input)
    }
```

--------------------------------

### Option::as_ref Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Converts from &Option<T> to Option<&T>.

```rust
let text: Option<String> = Some("Hello, world!".to_string());
// First, cast `Option<String>` to `Option<&String>` with `as_ref`,
// then consume *that* with `map`, leaving `text` on the stack.
let text_length: Option<usize> = text.as_ref().map(|s| s.len());
println!("still can print text: {text:?}");
```

--------------------------------

### url::Url::password() example

Source: https://docs.rs/url/latest/url/struct.Url.html

Returns the password as a percent-encoded ASCII string, if any. The example shows that an empty username (e.g., ftp://:secret123@example.com) still yields the password, and that URLs without a password return None.

```rust
use url::Url;

let url = Url::parse("ftp://rms:secret123@example.com")?;
assert_eq!(url.password(), Some("secret123"));

let url = Url::parse("ftp://:secret123@example.com")?;
assert_eq!(url.password(), Some("secret123"));

let url = Url::parse("ftp://rms@example.com")?;
assert_eq!(url.password(), None);

let url = Url::parse("https://example.com")?;
assert_eq!(url.password(), None);
```

--------------------------------

### Invalid URL example using url! macro

Source: https://docs.rs/url-macro/latest/url_macro

This example shows an invalid URL that triggers a compiler error when used with the url! macro. Use this to understand what happens when the URL is not valid.

```rust
// This triggers a compiler error
let invalid = url!("foo");
```

--------------------------------

### Full Rust example using url_encoded_data

Source: https://docs.rs/url_encoded_data/latest/url_encoded_data

Parses a query string with UrlEncodedData::parse_str and demonstrates as_pairs_of_original_order, as_map_of_single_key_to_multiple_values, as_map_of_single_key_to_first_occurrence_value, and as_map_of_single_key_to_last_occurrence_value. Uses the maplit crate for hashmap! macro. The example includes assertions to verify expected outputs and handles edge cases like keys without values and values without keys.

```rust
#[macro_use]
extern crate maplit;
use url_encoded_data::*;

fn main() {
    use std::borrow::Cow;
use url_encoded_data::UrlEncodedData;
let qs = "a=1&b=2&c=3&c=4&key_without_value&=value_without_key".to_string();
    for s in [
        qs.as_str(),
        ("https://abc.com/?".to_string() + qs.as_str()).as_str(),
        ("https://abc.com/?????".to_string() + qs.as_str()).as_str(),
    ]
        .iter()
    {
        let q = UrlEncodedData::parse_str(s);
        // let mut q = UrlEncodedData::prepare(url_1);
        // let q = q.parse();
        println!("got qs: {}", q);

        let pairs_expected_as_str = [
            ("a", "1"),
            ("b", "2"),
            ("c", "3"),
            ("c", "4"),
            ("key_without_value", ""),
            ("", "value_without_key"),
        ];

        for (i, (k, v)) in q.as_pairs_of_original_order().iter().enumerate() {
            let (k_, v_) = pairs_expected_as_str[i];
            assert_eq!(k.as_ref(), k_);
            assert_eq!(v.as_ref(), v_);
        }

        //
        let map_of_multiple_values_expected = hashmap! {
            "a"=>vec!("1"),
            "b"=>vec!("2"),
            "c"=>vec!("3", "4"),
            "key_without_value" => vec!(""),
            "" => vec!("value_without_key"),
        };
        dbg!("as_map_of_single_key_to_multiple_values");
        println!("as_map_of_single_key_to_multiple_values");
        let map = q.as_map_of_single_key_to_multiple_values();
        assert_eq!(map.len(), 5);

        for (k1, v1) in map {
            let v2 = map_of_multiple_values_expected.get(k1.as_ref()).unwrap();
            for (i, v2i) in v2.into_iter().enumerate() {
                assert_eq!(v1[i].to_string(), v2i.to_string());
            }
        }

        //
        let map_of_first_occurrence_value_expected = hashmap! {
            "a"=>"1",
            "b"=>"2",
            "c"=>"3",
            "key_without_value" => "",
            "" => "value_without_key",
        };
        dbg!("as_map_of_single_key_to_first_occurrence_value");
        let map = q.as_map_of_single_key_to_first_occurrence_value();
        assert_eq!(map.len(), 5);
        for (k1, v1) in map {
            let v2 = map_of_first_occurrence_value_expected
                .get(k1.as_ref())
                .unwrap();
            // let v3 = &v1;
            assert_eq!(&v1, v2); // ok, signifies comparing with references, it will auto-dereference to compare the value, which is more convenient
            let ptr1 = v1 as *const Cow<'_, str> as *const usize;
            let ptr2 = v2 as *const &str as *const usize;
            let msg = format!("{:p}, {:p}", ptr1, ptr2);
            dbg!(msg);
            println!("{:p}, {:p}", ptr1, ptr2);
            assert!(!std::ptr::eq(ptr1, ptr2));
            assert_eq!(*v1, **v2); // ok, value compare
        }

        //
        let map_of_last_occurrence_value_expected = hashmap! {
            "a"=>"1",
            "b"=>"2",
            "c"=>"4",
            "key_without_value" => "",
            "" => "value_without_key",
        };
        dbg!("as_map_of_single_key_to_last_occurrence_value");
        let map = q.as_map_of_single_key_to_last_occurrence_value();
        assert_eq!(map.len(), 5);

        for (k1, v1) in map {
            let v2 = map_of_last_occurrence_value_expected
                .get(k1.as_ref())
                .unwrap();
            assert_eq!(&v1, v2);
        }
    }
}
```

--------------------------------

### Custom Policies with PolicyBuilder

Source: https://docs.rs/url_jail/latest/url_jail

Shows how to build a custom policy using PolicyBuilder, starting from AllowPrivate and adding CIDR blocks and allows, plus a hostname block.

```rust
use url_jail::{PolicyBuilder, Policy};

let policy = PolicyBuilder::new(Policy::AllowPrivate)
    .block_cidr("10.0.0.0/8")           // Block specific range
    .allow_cidr("10.1.0.0/16")          // But allow a subnet
    .block_host("*.internal.example.com")
    .build();
```

--------------------------------

### Open a URL with url_open

Source: https://docs.rs/url_open/latest/url_open

Shows the minimal setup to open a URL in the default browser. Requires the url and url_open crates. Parses a URL and calls the open() method from the UrlOpen trait.

```rust
extern crate url;
extern crate url_open;

use url::Url;
use url_open::UrlOpen;

fn main() {
    Url::parse("https://github.com/overdrivenpotato/url_open").unwrap().open();
}
```

--------------------------------

### Configure URL parsing with Url::options

Source: https://docs.rs/url/latest/url/struct.Url.html

Returns a default ParseOptions to configure the URL parser. The example changes the base URL and parses a relative string.

```rust
use url::Url;
let options = Url::options();
let api = Url::parse("https://api.example.com")?;
let base_url = options.base_url(Some(&api));
let version_url = base_url.parse("version.json")?;
assert_eq!(version_url.as_str(), "https://api.example.com/version.json");
```

--------------------------------

### Retrieve URL Origin

Source: https://docs.rs/url/latest/src/url/lib.rs.html

Example of retrieving the origin for a URL with an ftp scheme.

```rust
use url::{Host, Origin, Url};
# use url::ParseError;

# fn run() -> Result<(), ParseError> {
let url = Url::parse("ftp://example.com/foo")?;
assert_eq!(url.origin(),
           Origin::Tuple("ftp".into(),
                         Host::Domain("example.com".into()),
```

--------------------------------

### Option::into_flat_iter Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Transforms an optional iterator into an iterator. Requires the option_into_flat_iter feature.

```rust
#![feature(option_into_flat_iter)]

let o1 = Some([1, 2]);
let o2 = None::<&[usize]>;

assert_eq!(o1.into_flat_iter().collect::<Vec<_>>(), [1, 2]);
assert_eq!(o2.into_flat_iter().collect::<Vec<_>>(), Vec::<&usize>::new());
```

--------------------------------

### Option::as_mut Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Converts from &mut Option<T> to Option<&mut T>.

```rust
let mut x = Some(2);
match x.as_mut() {
    Some(v) => *v = 42,
    None => {},
}
assert_eq!(x, Some(42));
```

--------------------------------

### Get File Host

Source: https://docs.rs/url/latest/src/url/parser.rs.html

Retrieves the host for a file URL, normalizing 'localhost' to an empty domain.

```rust
fn get_file_host(input: Input<'_>) -> ParseResult<(Host<Cow<'_, str>>, Input<'_>)> {
        let (_, host_str, remaining) = Parser::file_host(input)?;
        let host = match Host::parse(&host_str)? {
            Host::Domain(ref d) if d == "localhost" => Host::Domain(Cow::Borrowed("")),
            Host::Domain(s) => Host::Domain(Cow::Owned(s)),
            Host::Ipv4(ip) => Host::Ipv4(ip),
            Host::Ipv6(ip) => Host::Ipv6(ip),
        };
        Ok((host, remaining))
    }
```

--------------------------------

### Invalid URL macro example (compile error)

Source: https://docs.rs/url-static/latest/url_static

Shows a URL string with a space that will not compile because it is not a valid URL.

```rust
// ❌ This won't compile, because it'll never parse:
let woops = url!("https:// api.example.com");
```

--------------------------------

### url::Url::port() example

Source: https://docs.rs/url/latest/url/struct.Url.html

Returns the port number if explicitly specified. Default ports are not reflected; use port_or_known_default() to get defaults. The example shows that https://example.com:443/ returns None because 443 is the default for https.

```rust
use url::Url;

let url = Url::parse("https://example.com")?;
assert_eq!(url.port(), None);

let url = Url::parse("https://example.com:443/")?;
assert_eq!(url.port(), None);

let url = Url::parse("ssh://example.com:22")?;
assert_eq!(url.port(), Some(22));
```

--------------------------------

### Generate and print a QR code

Source: https://docs.rs/url2qr/latest/url2qr

Demonstrates how to generate a QR code matrix and a terminal-printable string for a given URL.

```rust
use url2qr::{generate_qr, qr_to_terminal};

let matrix = generate_qr("https://example.com").unwrap();
let output = qr_to_terminal("https://example.com", 4, false).unwrap();
println!("{output}");
```

--------------------------------

### Quick Start: Validate a URL with url_jail

Source: https://docs.rs/url_jail/latest/url_jail

Basic usage of the validate function with the PublicOnly policy. Returns a Validated struct containing the verified host and IP. The example uses the await keyword, indicating it is async.

```rust
use url_jail::{validate, Policy};

let result = validate("https://example.com/api", Policy::PublicOnly).await?;
println!("Safe to connect to {} ({})", result.host, result.ip);
```

--------------------------------

### Download SEC Annual Reports with urls2disk

Source: https://docs.rs/urls2disk/latest/urls2disk

This example demonstrates downloading Apple, Inc. annual reports as both raw HTML and converted PDF files. It shows how to configure the ClientBuilder with custom limits and wkhtmltopdf settings.

```rust
extern crate reqwest;
extern crate urls2disk;

use std::fs;
use std::path::Path;

use urls2disk::{wkhtmltopdf, ClientBuilder, Result, SimpleDocument, Url};

// This function will download Apple, Inc.'s annual reports for the years 2010 to 2017
// from the SEC's website to your disk.  It will download two copies of each annual
// report: one of just the raw html and another that has been converted to PDF.
fn run() -> Result<()> {
    // Create an output directory.
    let output_directory = Path::new("./data");
    if !output_directory.exists() {
        fs::create_dir_all(output_directory)?;
    }

    // Create a vector of urls we would like to download.
    // These urls represent the annual reports for Apple, Inc. from 2010 to 2017.
    let base = "https://www.sec.gov/Archives/edgar/data/";
    let urls = vec![
        "320193/000119312510238044/d10k.htm",
        "320193/000119312511282113/d220209d10k.htm",
        "320193/000119312512444068/d411355d10k.htm",
        "320193/000119312513416534/d590790d10k.htm",
        "320193/000119312514383437/d783162d10k.htm",
        "320193/000119312515356351/d17062d10k.htm",
        "320193/000162828016020309/a201610-k9242016.htm",
        "320193/000032019317000070/a10-k20179302017.htm",
    ].iter()
        .map(|stem| format!("{}{}", &base, stem))
        .collect::<Vec<String>>();

    // Turn the vector of urls into a vector of boxed Document trait objects (here we'll
    // be using the SimpleDocument struct as one possible implementer of the Document trait).
    // For this batch, we set the wkhtmltopdf option to false; so when we feed this list
    // to the Client it will just download the raw webpages in html format instead of
    // first converting them to PDF.
    let html_documents = urls.iter()
        .enumerate()
        .map(|(i, url_string)| {
            let filename = format!("Apple 10-K {}.html", i + 2010);
            let path = output_directory.join(&filename);
            let url = url_string.parse::<Url>()?;
            let wkhtmltopdf = false;
            let document = SimpleDocument::new(path, url, wkhtmltopdf);
            Ok(Box::new(document))
        })
        .collect::<Result<Vec<Box<SimpleDocument>>>>()?;

    // Turn the vector of urls into another vector of boxed Document trait objects
    // (to show off additional functionality).  This time we'll set the wkhtmltopdf
    // option to true; so when we feed this list to the Client it will first convert
    // the wepages to PDF before writing them to disk.
    let pdf_documents = urls.iter()
        .enumerate()
        .map(|(i, url_string)| {
            let filename = format!("Apple 10-K {}.pdf", i + 2010);
            let path = output_directory.join(&filename);
            let url = url_string.parse::<Url>()?;
            let wkhtmltopdf = true;
            let document = SimpleDocument::new(path, url, wkhtmltopdf);
            Ok(Box::new(document))
        })
        .collect::<Result<Vec<Box<SimpleDocument>>>>()?;

    // Combine our two vectors into one vector of Box<SimpleDocument>.
    let mut documents = [&html_documents[..], &pdf_documents[..]].concat();

    // Create the client.
    // Here, we're showing several customization options, but if you want to use
    // just the default settings, you could simply build the client with
    // `let client = ClientBuilder::default().build()?;`
    let client = ClientBuilder::default()
        .set_max_requests_per_second(9)
        .set_max_threads_cpu(4)
        .set_max_threads_io(50)
        .set_reqwest_client(reqwest::Client::new())
        .set_wkhtmltopdf_setting(wkhtmltopdf::Setting::Zoom(3.5))
        .set_wkhtmltopdf_settings(vec![
            wkhtmltopdf::Setting::DisableExternalLinks(true),
            wkhtmltopdf::Setting::DisableJavascript(true),
        ])
        .build()?;

    // Let the client go. It will download and write to disk all the
    // documents while simultaneously respecting the 'requests per second' and
    // other limits we provided. If you already have the documents on disk,
    // the client will not redownload them.
    client.get_documents(&mut documents)?;

    // Note: Here, if you want to, you can now access the raw bytes of all the urls
    // you downloaded, since they are now stored on each SimpleDocument in addition
    // to being saved on your disk.
    Ok(())
}

fn main() {
    run().unwrap();
}
```

--------------------------------

### Join relative URLs with Url::join

Source: https://docs.rs/url/latest/src/url/lib.rs.html

Resolves a relative URL string against a base URL. A trailing slash on the base is significant: without it, the last path component is treated as a file name and removed. A scheme-relative special URL (starting with //) replaces everything after the scheme, and an absolute URL replaces the entire base URL.

```rust
use url::Url;
# use url::ParseError;

// Base without a trailing slash
# fn run() -> Result<(), ParseError> {
let base = Url::parse("https://example.net/a/b.html")?;
let url = base.join("c.png")?;
assert_eq!(url.as_str(), "https://example.net/a/c.png");  // Not /a/b.html/c.png

// Base with a trailing slash
let base = Url::parse("https://example.net/a/b/")?;
let url = base.join("c.png")?;
assert_eq!(url.as_str(), "https://example.net/a/b/c.png");

// Input as scheme relative special URL
let base = Url::parse("https://alice.com/a")?;
let url = base.join("//eve.com/b")?;
```

--------------------------------

### Create short hashes and test prefix relationships

Source: https://docs.rs/url-hash/latest/url_hash

Shows how to obtain UrlShortHash and UrlVeryShortHash from a UrlHash, and how to use starts_with and starts_with_just to test prefix relationships. Assumes a Url is parsed and a UrlHash is created.

```rust
let url = Url::parse("https://doc.rust-lang.org/").unwrap();
let hash =  UrlHash::from(url);

let short = hash.short();
assert!(hash.starts_with(&short));

let very_short = hash.very_short();
assert!(short.starts_with(&very_short));
assert!(hash.starts_with_just(&very_short));

assert_eq!(very_short, hash.very_short());
```

--------------------------------

### path

Source: https://docs.rs/url/latest/url/struct.Url.html

Returns the path of the URL as a percent-encoded ASCII string. For cannot-be-a-base URLs, it returns an arbitrary string not starting with '/', otherwise it starts with '/'.

```APIDOC
## path

### Description
Returns the path for this URL as a percent-encoded ASCII string. For cannot-be-a-base URLs, this is an arbitrary string that doesn't start with '/'. For other URLs, this starts with a '/' slash and continues with slash-separated path segments.

### Method
pub fn path(&self) -> &str

### Return Value
- **&str** - The path as a percent-encoded ASCII string.

### Examples
```rust
use url::{Url, ParseError};

let url = Url::parse("https://example.com/api/versions?page=2")?;
assert_eq!(url.path(), "/api/versions");

let url = Url::parse("https://example.com")?;
assert_eq!(url.path(), "/");

let url = Url::parse("https://example.com/countries/việt nam")?;
assert_eq!(url.path(), "/countries/vi%E1%BB%87t%20nam");
```
```

--------------------------------

### Option::replace example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Replaces the value in an Option, returning the old value. The example shows that replace returns the previous value and sets the new one.

```rust
let mut x = None;
let old = x.replace(3);
assert_eq!(x, Some(3));
assert_eq!(old, None);
```

--------------------------------

### url::Url::has_host() example

Source: https://docs.rs/url/latest/url/struct.Url.html

Returns true if the URL has a host, equivalent to url.host().is_some(). The example shows that ftp URLs with a host return true, while unix and data URLs return false.

```rust
use url::Url;

let url = Url::parse("ftp://rms@example.com")?;
assert!(url.has_host());

let url = Url::parse("unix:/run/foo.socket")?;
assert!(!url.has_host());

let url = Url::parse("data:text/plain,Stuff")?;
assert!(!url.has_host());
```

--------------------------------

### Substitute URL placeholders with UrlTemplate

Source: https://docs.rs/urltemplate/latest/urltemplate

Shows how to create a UrlTemplate from a URL with placeholders and substitute values from a HashMap. The example uses the substitute_str method and asserts the resulting URL. Note the placeholder syntax {source} and the use of a HashMap for parameters.

```rust
extern crate urltemplate;

use urltemplate::UrlTemplate;
use std::collections::HashMap;

let mut params = HashMap::new();
params.insert("source".to_string(), "url-template-crate-❤".to_string());
let url_with_placeholders = UrlTemplate::from("https://www.mozilla.org/?utm_source={source}");
let url =  url_with_placeholders.substitute_str(&params).expect("valid url");
assert_eq!(url, "https://www.mozilla.org/?utm_source=url-template-crate-❤")
```

--------------------------------

### Parse Host Entry Point

Source: https://docs.rs/url/latest/src/url/parser.rs.html

Initializes host parsing, delegating to file-specific logic if the scheme type is a file.

```rust
991    pub fn parse_host(
992        mut input: Input<'_>,
993        scheme_type: SchemeType,
994    ) -> ParseResult<(Host<Cow<'_, str>>, Input<'_>)> {
995        if scheme_type.is_file() {
996            return Parser::get_file_host(input);
997        }
```

--------------------------------

### Build a URL with URLBuilder

Source: https://docs.rs/url-builder/latest/url_builder

Constructs a URL with protocol, host, port, and query parameters, then prints the built URL. The order of query parameters is indeterminate because they are stored in a HashMap.

```rust
use url_builder::URLBuilder;

let mut ub = URLBuilder::new();

ub.set_protocol("http")
    .set_host("localhost")
    .set_port(8000)
    .add_param("first", "1")
    .add_param("second", "2")
    .add_param("third", "3");

println!("{}", ub.build());
```

--------------------------------

### Try all URL shorteners until one succeeds (try_generate)

Source: https://docs.rs/urlshortener/latest/urlshortener

Creates a UrlShortener client and attempts to generate a short URL using all available providers until one succeeds. The result is checked with assert! that it is Ok.

```rust
use urlshortener::client::UrlShortener;

let us = UrlShortener::new().unwrap();
let short_url = us.try_generate("https://my-long-url.com", None);
assert!(short_url.is_ok());
```

--------------------------------

### Option::is_some Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Checks if an option is a Some value.

```rust
let x: Option<u32> = Some(2);
assert_eq!(x.is_some(), true);

let x: Option<u32> = None;
assert_eq!(x.is_some(), false);
```

--------------------------------

### ok_or usage

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Demonstrates transforming an Option into a Result using ok_or.

```rust
let x = Some("foo");
assert_eq!(x.ok_or(0), Ok("foo"));

let x: Option<&str> = None;
assert_eq!(x.ok_or(0), Err(0));
```

--------------------------------

### Option::is_none Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Checks if an option is a None value.

```rust
let x: Option<u32> = Some(2);
assert_eq!(x.is_none(), false);

let x: Option<u32> = None;
assert_eq!(x.is_none(), true);
```

--------------------------------

### Option::is_some_and Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Checks if an option is Some and matches a predicate.

```rust
let x: Option<u32> = Some(2);
assert_eq!(x.is_some_and(|x| x > 1), true);

let x: Option<u32> = Some(0);
assert_eq!(x.is_some_and(|x| x > 1), false);

let x: Option<u32> = None;
assert_eq!(x.is_some_and(|x| x > 1), false);

let x: Option<String> = Some("ownership".to_string());
assert_eq!(x.as_ref().is_some_and(|x| x.len() > 1), true);
println!("still alive {:?}", x);
```

--------------------------------

### Create a relative URL

Source: https://docs.rs/url/latest/src/url/lib.rs.html

Demonstrates how to use the make_relative method to find the relative path from a base URL to a target URL.

```rust
use url::Url;
# use url::ParseError;

# fn run() -> Result<(), ParseError> {
let base = Url::parse("https://example.net/a/b.html")?;
let url = Url::parse("https://example.net/a/c.png")?;
let relative = base.make_relative(&url);
assert_eq!(relative.as_ref().map(|s| s.as_str()), Some("c.png"));

let base = Url::parse("https://example.net/a/b/")?;
let url = Url::parse("https://example.net/a/b/c.png")?;
let relative = base.make_relative(&url);
assert_eq!(relative.as_ref().map(|s| s.as_str()), Some("c.png"));

let base = Url::parse("https://example.net/a/b/")?;
let url = Url::parse("https://example.net/a/d/c.png")?;
let relative = base.make_relative(&url);
assert_eq!(relative.as_ref().map(|s| s.as_str()), Some("../d/c.png"));

let base = Url::parse("https://example.net/a/b.html?c=d")?;
let url = Url::parse("https://example.net/a/b.html?e=f")?;
let relative = base.make_relative(&url);
assert_eq!(relative.as_ref().map(|s| s.as_str()), Some("?e=f"));
# Ok(())
# }
# run().unwrap();
```

--------------------------------

### Get URL pathname

Source: https://docs.rs/url/latest/src/url/quirks.rs.html

Retrieves the path component of the URL.

```rust
/// Getter for <https://url.spec.whatwg.org/#dom-url-pathname>
#[inline]
pub fn pathname(url: &Url) -> &str {
    url.path()
}
```

--------------------------------

### Get URL port

Source: https://docs.rs/url/latest/src/url/quirks.rs.html

Retrieves the port component of the URL.

```rust
/// Getter for <https://url.spec.whatwg.org/#dom-url-port>
#[inline]
pub fn port(url: &Url) -> &str {
    &url[Position::BeforePort..Position::AfterPort]
}
```

--------------------------------

### Generate short URL with specified provider (Provider::IsGd)

Source: https://docs.rs/urlshortener/latest/urlshortener

Creates a UrlShortener client and generates a short URL using the IsGd provider. The result is checked with assert! that it is Ok.

```rust
use urlshortener::{providers::Provider, client::UrlShortener};

let us = UrlShortener::new().unwrap();
let short_url = us.generate("https://my-long-url.com", &Provider::IsGd);
assert!(short_url.is_ok());
```

--------------------------------

### Get URL hostname

Source: https://docs.rs/url/latest/src/url/quirks.rs.html

Retrieves the hostname component of the URL.

```rust
/// Getter for <https://url.spec.whatwg.org/#dom-url-hostname>
#[inline]
pub fn hostname(url: &Url) -> &str {
    url.host_str().unwrap_or("")
}
```

--------------------------------

### impl<T, U> Product<Option<U>> for Option<T>

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Takes each element in the iterator: if it is a None, no further elements are taken, and the None is returned. Should no None occur, the product of all elements is returned.

```APIDOC
## impl<T, U> Product<Option<U>> for Option<T>

### Description
Takes each element in the iterator: if it is a None, no further elements are taken, and the None is returned. Should no None occur, the product of all elements is returned.

### Method
fn product<I>(iter: I) -> Option<T> where I: Iterator<Item = Option<U>>

### Examples
```rust
let nums = vec!["5", "10", "1", "2"];
let total: Option<usize> = nums.iter().map(|w| w.parse::<usize>().ok()).product();
assert_eq!(total, Some(100));
let nums = vec!["5", "10", "one", "2"];
let total: Option<usize> = nums.iter().map(|w| w.parse::<usize>().ok()).product();
assert_eq!(total, None);
```
```

--------------------------------

### Option::is_none_or Example

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Checks if an option is None or matches a predicate.

```rust
let x: Option<u32> = Some(2);
assert_eq!(x.is_none_or(|x| x > 1), true);

let x: Option<u32> = Some(0);
assert_eq!(x.is_none_or(|x| x > 1), false);

let x: Option<u32> = None;
assert_eq!(x.is_none_or(|x| x > 1), true);

let x: Option<String> = Some("ownership".to_string());
assert_eq!(x.as_ref().is_none_or(|x| x.len() > 1), true);
println!("still alive {:?}", x);
```

--------------------------------

### origin: Get URL origin for other schemes

Source: https://docs.rs/url/latest/url/struct.Url.html

For non-special schemes, the origin is not a tuple.

```rust
use url::{Host, Origin, Url};

let url = Url::parse("foo:bar")?;
assert!(!url.origin().is_tuple());
```

--------------------------------

### Generate short URL with authenticated provider (Provider::GooGl)

Source: https://docs.rs/urlshortener/latest/urlshortener

Creates a UrlShortener client and generates a short URL using the GooGl provider with an API key. The result is checked with assert! that it is Ok.

```rust
use urlshortener::{ client::UrlShortener, providers::Provider };

let us = UrlShortener::new().unwrap();
let key = "MY_API_KEY";
let short_url = us.generate("https://my-long-url.com", &Provider::GooGl { api_key:
key.to_owned() });
assert!(short_url.is_ok());
```

--------------------------------

### Using and

Source: https://docs.rs/url/latest/url/type.EncodingOverride.html

Returns None if the option is None, otherwise returns the provided option.

```rust
let x = Some(2);
let y: Option<&str> = None;
assert_eq!(x.and(y), None);

let x: Option<u32> = None;
let y = Some("foo");
assert_eq!(x.and(y), None);

let x = Some(2);
let y = Some("foo");
assert_eq!(x.and(y), Some("foo"));

let x: Option<u32> = None;
let y: Option<&str> = None;
assert_eq!(x.and(y), None);
```

--------------------------------

### URL origin with file scheme

Source: https://docs.rs/url/latest/src/url/lib.rs.html

Demonstrates that file URLs do not have a tuple origin and that distinct file URLs have different origins.

```rust
use url::{Host, Origin, Url};
# use url::ParseError;

# fn run() -> Result<(), ParseError> {
let url = Url::parse("file:///tmp/foo")?;
assert!(!url.origin().is_tuple());

let other_url = Url::parse("file:///tmp/foo")?;
assert!(url.origin() != other_url.origin());
# Ok(())
# }
# run().unwrap();
```

--------------------------------

### Create a UrlHash from a Url and print it

Source: https://docs.rs/url-hash/latest/url_hash

Parses a URL and creates a UrlHash from it, then prints the hash. Uses the url crate's Url::parse and url_hash::UrlHash.

```rust
use url::Url;
use url_hash::UrlHash;

let url = Url::parse("https://doc.rust-lang.org/").unwrap();
let hash =  UrlHash::from(url);
println!("{}", hash);
```

--------------------------------

### Benefits of normalization strings for significant URL parts

Source: https://docs.rs/urlnorm/latest/urlnorm

The page presents several examples of URL normalization behavior, likely to illustrate the output of compute_normalization_string or a similar method, but only the text results are given. The snippet shows the normalized strings for various URLs, demonstrating host and path normalization, query sorting, and fragment dropping.

```text
http://efekarakus.github.io/twitch-analytics/#/revenue
efekarakus.github.io:twitch-analytics:revenue:

http://fusion.net/story/121315/maybe-crickets-arent-the-food-of-the-future-after-all/?utm_source=facebook&utm_medium=social&utm_campaign=quartz
fusion.net:story:121315:maybe-crickets-arent-the-food-of-th-future-after-all:

http://www.capradio.org/news/npr/story?storyid=382276026
capradio.org:news:npr:story:storyid:382276026:

http://www.charlotteobserver.com/2015/02/23/5534630/charlotte-city-council-proposals.html#.VOxrajTF91E
charlotteobserver.com:2015:02:23:5534630:charlotte-city-council-proposals:

http://www.m.webmd.com/melanoma-skin-cancer/news/20150409/fewer-us-children-getting-melanoma-study?src=RSS_PUBLIC
webmd.com:melanoma-skin-cancer:news:20150409:fewer-us-children-getting-melanoma-study:src:RSS_PUBLIC:
```

--------------------------------

### scheme: Get URL scheme

Source: https://docs.rs/url/latest/url/struct.Url.html

Returns the scheme of the URL, lower-cased, without the ':' delimiter.

```rust
use url::Url;

let url = Url::parse("file:///tmp/foo")?;
assert_eq!(url.scheme(), "file");
```
Tokens:
Raw