Pingora Header Serde

https://docs.rs/pingora-header-serde/0.8.0
Admin
Pingora Header Serde is a Rust crate that serializes HTTP response headers with compression,...
Tokens:16,683
Snippets:90
Trust Score:9.7
Update:7 months ago
Show doc for...
Tokens:
Raw
### Get Zstd Frame Content Size

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html

Retrieves the content size of a Zstandard compressed frame from a byte slice. It returns 'invalid', 'unknown', 'none', or the actual size as a string, handling potential errors during size retrieval.

```rust
fn get_frame_content_size(source: &[u8]) -> ImmutStr {
    match zstd_safe::get_frame_content_size(source) {
        Ok(Some(size)) => match size {
            zstd_safe::CONTENTSIZE_ERROR => ImmutStr::from("invalid"),
            zstd_safe::CONTENTSIZE_UNKNOWN => ImmutStr::from("unknown"),
            _ => ImmutStr::from(size.to_string()),
        },
        Ok(None) => ImmutStr::from("none"),
        Err(_e) => ImmutStr::from("failed"),
    }
}
```

--------------------------------

### HeaderSerde Initialization

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=std%3A%3Avec

Methods for creating a new HeaderSerde instance with optional compression dictionary support.

```APIDOC
## POST /HeaderSerde/new

### Description
Creates a new instance of HeaderSerde. An optional zstd compression dictionary can be provided to improve the compression ratio and speed.

### Method
POST

### Parameters
#### Request Body
- **dict** (Option<Vec<u8>>) - Optional - An optional zstd compression dictionary for optimized serialization.

### Request Example
{
  "dict": [0, 1, 2, 3]
}

### Response
#### Success Response (200)
- **instance** (HeaderSerde) - The initialized HeaderSerde object.
```

--------------------------------

### HeaderSerde Initialization

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=u32+-%3E+bool

Creates a new instance of HeaderSerde with an optional zstd compression dictionary.

```APIDOC
## Constructor: HeaderSerde::new

### Description
Initializes a new HeaderSerde instance. An optional zstd compression dictionary can be provided to optimize compression ratio and performance.

### Parameters
#### Request Body
- **dict** (Option<Vec<u8>>) - Optional - A zstd compression dictionary for optimized serialization.

### Response
- **HeaderSerde** (Struct) - The initialized serialization handler.
```

--------------------------------

### Initialize and Use HeaderSerde for Serialization and Deserialization

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=

Demonstrates how to instantiate the HeaderSerde struct with an optional dictionary and perform serialization of a ResponseHeader or deserialization of binary data back into a header object.

```rust
use pingora_header_serde::HeaderSerde;

// Initialize with an optional zstd dictionary
let serde = HeaderSerde::new(None);

// Serialize a ResponseHeader to bytes
let serialized_data = serde.serialize(&response_header)?;

// Deserialize bytes back to a ResponseHeader
let deserialized_header = serde.deserialize(&serialized_data)?;
```

--------------------------------

### Initialize HeaderSerde with Zstd Compression

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html?search=std%3A%3Avec

Creates a new HeaderSerde instance, optionally using a zstd compression dictionary for improved compression ratio and speed. The dictionary can be provided as a Vec<u8>.

```rust
pub fn new(dict: Option<Vec<u8>>) -> Self {
    if let Some(dict) = dict {
        HeaderSerde {
            compression: ZstdCompression::WithDict(thread_zstd::CompressionWithDict::new(
                &dict,
                COMPRESS_LEVEL,
            )),
            buf: ThreadLocal::new(),
        }
    } else {
        HeaderSerde {
            compression: ZstdCompression::Default(
                thread_zstd::Compression::new(),
                COMPRESS_LEVEL,
            ),
            buf: ThreadLocal::new(),
        }
    }
}
```

--------------------------------

### Initialize and Use HeaderSerde for Serialization

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Demonstrates how to instantiate the HeaderSerde struct with an optional zstd dictionary and perform serialization and deserialization of ResponseHeader objects.

```rust
use pingora_header_serde::HeaderSerde;

// Initialize with an optional zstd dictionary
let dict = Some(vec![0u8; 1024]);
let serde = HeaderSerde::new(dict);

// Serialize a ResponseHeader
let serialized = serde.serialize(&response_header)?;

// Deserialize back into a ResponseHeader
let deserialized = serde.deserialize(&serialized)?;
```

--------------------------------

### Initialize and Use HeaderSerde for Serialization

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html

Demonstrates how to instantiate the HeaderSerde struct with an optional zstd dictionary and perform serialization and deserialization of HTTP response headers.

```rust
use pingora_header_serde::HeaderSerde;

// Initialize with an optional zstd dictionary
let serde = HeaderSerde::new(None);

// Serialize a ResponseHeader
let serialized_data = serde.serialize(&header)?;

// Deserialize back to a ResponseHeader
let deserialized_header = serde.deserialize(&serialized_data)?;
```

--------------------------------

### Initialize and Use HeaderSerde for Compression

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html?search=u32+-%3E+bool

The HeaderSerde struct manages the compression lifecycle using thread-local buffers. It supports initialization with an optional dictionary and provides methods to serialize and deserialize HTTP response headers.

```rust
pub fn new(dict: Option<Vec<u8>>) -> Self {
    if let Some(dict) = dict {
        HeaderSerde {
            compression: ZstdCompression::WithDict(thread_zstd::CompressionWithDict::new(
                &dict,
                COMPRESS_LEVEL,
            )),
            buf: ThreadLocal::new(),
        }
    } else {
        HeaderSerde {
            compression: ZstdCompression::Default(
                thread_zstd::Compression::new(),
                COMPRESS_LEVEL,
            ),
            buf: ThreadLocal::new(),
        }
    }
}

pub fn serialize(&self, header: &ResponseHeader) -> Result<Vec<u8>> {
    let mut buf = self
        .buf
        .get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE)))
        .borrow_mut();
    buf.clear();
    resp_header_to_buf(header, &mut buf);
    self.compression.compress(&buf)
}

pub fn deserialize(&self, data: &[u8]) -> Result<ResponseHeader> {
    let mut buf = self
        .buf
        .get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE)))
        .borrow_mut();
    buf.clear();
    self.compression
        .decompress_to_buffer(data, buf.deref_mut())?;
    buf_to_http_header(&buf)
}
```

--------------------------------

### Create HeaderSerde Instance in Rust

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=std%3A%3Avec

Constructs a new HeaderSerde instance. An optional zstd compression dictionary can be provided to potentially enhance compression efficiency and speed. This function is essential for initializing the header serialization/deserialization utility.

```rust
pub fn new(dict: Option<Vec<u8>>) -> Self
```

--------------------------------

### Serialize HTTP Response Header with/without Dictionary (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/dict.rs.html

Demonstrates the serialization of a `ResponseHeader` using `HeaderSerde`. It compares the compressed size of a header serialized with a pre-trained zstd dictionary against one serialized without a dictionary and against the uncompressed representation. This highlights the compression benefits of using a dictionary.

```Rust
use super::*;
use crate::resp_header_to_buf;
use pingora_http::ResponseHeader;

fn gen_test_dict() -> Vec<u8> {
    let mut path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
    path.push("samples/test");
    train(path)
}

fn gen_test_header() -> ResponseHeader {
    let mut header = ResponseHeader::build(200, None).unwrap();
    header
        .append_header("Date", "Thu, 23 Dec 2021 11:23:29 GMT")
        .unwrap();
    header
        .append_header("Last-Modified", "Sat, 09 Oct 2021 22:41:34 GMT")
        .unwrap();
    header.append_header("Connection", "keep-alive").unwrap();
    header.append_header("Vary", "Accept-encoding").unwrap();
    header.append_header("Content-Encoding", "gzip").unwrap();
    header
        .append_header("Access-Control-Allow-Origin", "*")
        .unwrap();
    header
}

#[test]
fn test_ser_with_dict() {
    let dict = gen_test_dict();
    let serde = crate::HeaderSerde::new(Some(dict));
    let serde_no_dict = crate::HeaderSerde::new(None);
    let header = gen_test_header();

    let compressed = serde.serialize(&header).unwrap();
    let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();
    let mut buf = vec![];
    let uncompressed = resp_header_to_buf(&header, &mut buf);

    assert!(compressed.len() < uncompressed);
    assert!(compressed.len() < compressed_no_dict.len());
}
```

--------------------------------

### Serialize and Deserialize HTTP Headers with Zstd Dictionary in Rust

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/dict.rs.html?search=u32+-%3E+bool

Demonstrates the serialization and deserialization of HTTP ResponseHeaders using the HeaderSerde struct. It shows how to use a zstd dictionary for compression, resulting in smaller serialized data compared to not using a dictionary or using uncompressed data. Tests verify the size reduction and data integrity.

```rust
    #[test]
    fn test_ser_with_dict() {
        let dict = gen_test_dict();
        let serde = crate::HeaderSerde::new(Some(dict));
        let serde_no_dict = crate::HeaderSerde::new(None);
        let header = gen_test_header();

        let compressed = serde.serialize(&header).unwrap();
        let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();
        let mut buf = vec![];
        let uncompressed = resp_header_to_buf(&header, &mut buf);

        assert!(compressed.len() < uncompressed);
        assert!(compressed.len() < compressed_no_dict.len());
    }

    #[test]
    fn test_deserialize_with_dict() {
        let dict = gen_test_dict();
        let serde = crate::HeaderSerde::new(Some(dict));
        let serde_no_dict = crate::HeaderSerde::new(None);
        let header = gen_test_header();

        let compressed = serde.serialize(&header).unwrap();
        let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();

        let from_dict_header = serde.deserialize(&compressed).unwrap();
        let from_no_dict_header = serde_no_dict.deserialize(&compressed_no_dict).unwrap();

        assert_eq!(from_dict_header.status, from_no_dict_header.status);
        assert_eq!(from_dict_header.headers, from_no_dict_header.headers);
    }

    #[test]
    fn test_ser_de_with_dict() {
        let dict = gen_test_dict();
        let serde = crate::HeaderSerde::new(Some(dict));
        let header = gen_test_header();

        let compressed = serde.serialize(&header).unwrap();
        let header2 = serde.deserialize(&compressed).unwrap();

        assert_eq!(header.status, header2.status);
        assert_eq!(header.headers, header2.headers);
    }

```

--------------------------------

### Rust: Zstd Compression with Dictionary

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html

Enables Zstd compression and decompression using a pre-defined dictionary for potentially better compression ratios. It maintains thread-local contexts and owns the compression and decompression dictionaries. The `compress` method takes raw data and returns compressed data. Dependencies include `zstd_safe` and `thread_local`.

```rust
use std::cell::{RefCell, RefMut};
use thread_local::ThreadLocal;
use zstd_safe::{CCtx, CDict, DCtx, DDict};

pub struct CompressionWithDict {
    inner: CompressionInner,
    com_dict: CDict<'static>,
    de_dict: DDict<'static>,
}

impl CompressionWithDict {
    pub fn new(dict: &[u8], compression_level: i32) -> Self {
        CompressionWithDict {
            inner: CompressionInner::new(),
            com_dict: CDict::create(dict, compression_level),
            de_dict: DDict::create(dict),
        }
    }

    pub fn compress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
    ) -> Result<usize, &'static str> {
        self.inner
            .compress_to_buffer_using_dict(source, destination, &self.com_dict)
    }

    pub fn compress(&self, data: &[u8]) -> Result<Vec<u8>, &'static str> {
        let mut buffer = make_compressed_data_buffer(data.len());
        self.compress_to_buffer(data, &mut buffer)?;
        Ok(buffer)
    }

    pub fn decompress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
    ) -> Result<usize, &'static str> {
        self.inner
            .decompress_to_buffer_using_dict(source, destination, &self.de_dict)
    }
}

// Assuming CompressionInner and its methods are defined as in the previous snippet
// and that `compress_to_buffer_using_dict` and `decompress_to_buffer_using_dict` exist.

// Placeholder for CompressionInner and its relevant methods
struct CompressionInner {
    com_context: ThreadLocal<RefCell<zstd_safe::CCtx<'static>>>,
    de_context: ThreadLocal<RefCell<zstd_safe::DCtx<'static>>>,
}

impl CompressionInner {
    fn new() -> Self {
        CompressionInner {
            com_context: ThreadLocal::new(),
            de_context: ThreadLocal::new(),
        }
    }

    #[inline]
    fn get_com_context(&self) -> RefMut<'_, CCtx<'static>> {
        self.com_context
            .get_or(|| RefCell::new(CCtx::create()))
            .borrow_mut()
    }

    #[inline]
    fn get_de_context(&self) -> RefMut<'_, DCtx<'static>> {
        self.de_context
            .get_or(|| RefCell::new(DCtx::create()))
            .borrow_mut()
    }

    fn compress_to_buffer_using_dict<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
        dict: &CDict<'static>,
    ) -> Result<usize, &'static str> {
        self.get_com_context()
            .compress_with_dict(destination, source, dict)
            .map_err(zstd_safe::get_error_name)
    }

    fn decompress_to_buffer_using_dict<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
        dict: &DDict<'static>,
    ) -> Result<usize, &'static str> {
        self.get_de_context()
            .decompress_with_dict(destination, source, dict)
            .map_err(zstd_safe::get_error_name)
    }
}

// Placeholder for make_compressed_data_buffer
fn make_compressed_data_buffer(size: usize) -> Vec<u8> {
    Vec::with_capacity(size)
}

```

--------------------------------

### Train Zstd Dictionary from Directory (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/dict/fn.train.html

Trains a zstd dictionary from all files within a specified directory path. The function takes a path as input and returns the trained dictionary as a byte vector. This is useful for optimizing compression of multiple files with similar content.

```rust
pub fn train<P: AsRef<Path>>(dir_path: P) -> Vec<u8> ⓘ
```

--------------------------------

### Initialize and Use HeaderSerde for HTTP Header Compression

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html

The HeaderSerde struct provides methods to serialize and deserialize HTTP response headers. It supports optional Zstd dictionaries for improved compression and uses thread-local storage to manage internal buffers efficiently.

```rust
pub struct HeaderSerde {
    compression: ZstdCompression,
    buf: ThreadLocal<RefCell<Vec<u8>>>,
}

impl HeaderSerde {
    pub fn new(dict: Option<Vec<u8>>) -> Self { /* ... */ }

    pub fn serialize(&self, header: &ResponseHeader) -> Result<Vec<u8>> {
        let mut buf = self.buf.get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE))).borrow_mut();
        buf.clear();
        resp_header_to_buf(header, &mut buf);
        self.compression.compress(&buf)
    }

    pub fn deserialize(&self, data: &[u8]) -> Result<ResponseHeader> {
        let mut buf = self.buf.get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE))).borrow_mut();
        buf.clear();
        self.compression.decompress_to_buffer(data, buf.deref_mut())?;
        buf_to_http_header(&buf)
    }
}
```

--------------------------------

### Initialize and Use HeaderSerde for HTTP Header Compression

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

The HeaderSerde struct provides methods to serialize and deserialize HTTP response headers. It manages internal buffers using thread-local storage and supports both default and dictionary-based Zstd compression.

```rust
pub struct HeaderSerde {
    compression: ZstdCompression,
    buf: ThreadLocal<RefCell<Vec<u8>>>,
}

impl HeaderSerde {
    pub fn new(dict: Option<Vec<u8>>) -> Self {
        // Initialization logic for compression with or without dictionary
    }

    pub fn serialize(&self, header: &ResponseHeader) -> Result<Vec<u8>> {
        let mut buf = self.buf.get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE))).borrow_mut();
        buf.clear();
        resp_header_to_buf(header, &mut buf);
        self.compression.compress(&buf)
    }

    pub fn deserialize(&self, data: &[u8]) -> Result<ResponseHeader> {
        let mut buf = self.buf.get_or(|| RefCell::new(Vec::with_capacity(MAX_HEADER_BUF_SIZE))).borrow_mut();
        buf.clear();
        self.compression.decompress_to_buffer(data, buf.deref_mut())?;
        buf_to_http_header(&buf)
    }
}
```

--------------------------------

### HeaderSerde API

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

This section details the methods available for the HeaderSerde struct, including creation, serialization, and deserialization of HTTP response headers.

```APIDOC
## Struct HeaderSerde

### Description
HTTP Response header serialization. This struct provides the APIs to convert HTTP response header into compressed wired format for storage.

## Implementations

### impl HeaderSerde

#### pub fn new(dict: Option<Vec<u8>>) -> Self

##### Description
Create a new HeaderSerde. An optional zstd compression dictionary can be provided to improve the compression ratio and speed. See dict for more details.

##### Method
`new`

##### Parameters
- `dict` (Option<Vec<u8>>) - Optional - A zstd compression dictionary.

#### pub fn serialize(&self, header: &ResponseHeader) -> Result<Vec<u8>>

##### Description
Serialize the given response header.

##### Method
`serialize`

##### Parameters
- `header` (&ResponseHeader) - Required - The response header to serialize.

##### Returns
- `Result<Vec<u8>>` - A Result containing the serialized data or an error.

#### pub fn deserialize(&self, data: &[u8]) -> Result<ResponseHeader>

##### Description
Deserialize the given response header.

##### Method
`deserialize`

##### Parameters
- `data` (&[u8]) - Required - The byte slice containing the serialized header data.

##### Returns
- `Result<ResponseHeader>` - A Result containing the deserialized ResponseHeader or an error.
```

--------------------------------

### Implement Thread-Local Zstandard Compression

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html?search=

Provides structures for standard and dictionary-based Zstandard compression. These wrappers manage thread-local contexts to ensure efficient and thread-safe data processing.

```rust
use std::cell::{RefCell, RefMut};
use thread_local::ThreadLocal;
use zstd_safe::{CCtx, CDict, DCtx, DDict};

#[derive(Default)]
pub struct Compression(CompressionInner);

impl Compression {
    pub fn compress(&self, data: &[u8], level: i32) -> Result<Vec<u8>, &'static str> {
        let mut buffer = make_compressed_data_buffer(data.len());
        self.compress_to_buffer(data, &mut buffer, level)?;
        Ok(buffer)
    }
}

#[derive(Default)]
struct CompressionInner {
    com_context: ThreadLocal<RefCell<zstd_safe::CCtx<'static>>>,
    de_context: ThreadLocal<RefCell<zstd_safe::DCtx<'static>>>,
}

impl CompressionInner {
    fn get_com_context(&self) -> RefMut<'_, CCtx<'static>> {
        self.com_context
            .get_or(|| RefCell::new(CCtx::create()))
            .borrow_mut()
    }
}
```

--------------------------------

### Type Conversion with try_from (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/dict/fn.train.html?search=u32+-%3E+bool

Demonstrates the 'try_from' method for the HeaderSerde type. This method allows for type conversion where 'u32' is matched against a generic type 'U' and 'bool' is matched against a generic type 'T', returning a Result. It's used for safe conversions between different data types.

```Rust
pingora_header_serde::HeaderSerde::try_from
**U** -> Result<**T** >
where
u32 matches **U**
bool matches **T**
```

--------------------------------

### Compress Data to Buffer Using Dictionary (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html?search=u32+-%3E+bool

Compresses source data into a destination buffer using a provided compression dictionary. Returns the number of bytes written or an error string. Requires a mutable WriteBuf for the destination and a CDict for the dictionary.

```rust
fn compress_to_buffer_using_dict<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
        dict: &CDict,
    ) -> Result<usize, &'static str> {
        self.get_com_context()
            .compress_using_cdict(destination, source, dict)
            .map_err(zstd_safe::get_error_name)
    }
```

--------------------------------

### Train Zstd Dictionary from Files in Rust

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/dict.rs.html?search=u32+-%3E+bool

This function trains a zstd dictionary from a directory of files. It reads all files within the specified directory and uses them to generate a dictionary, which can then be used for more efficient compression. The maximum dictionary size is set to 64MB.

```rust
// Copyright 2026 Cloudflare, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//! Training to generate the zstd dictionary.

use std::fs;
use zstd::dict;

/// Train the zstd dictionary from all the files under the given `dir_path`
///
/// The output will be the trained dictionary
pub fn train<P: AsRef<std::path::Path>>(dir_path: P) -> Vec<u8> {
    // TODO: check f is file, it can be dir
    let files = fs::read_dir(dir_path)
        .unwrap()
        .filter_map(|entry| entry.ok().map(|f| f.path()));
    dict::from_files(files, 64 * 1024 * 1024).unwrap()
}

#[cfg(test)]
mod test {
    use super::*;
    use crate::resp_header_to_buf;
    use pingora_http::ResponseHeader;

    fn gen_test_dict() -> Vec<u8> {
        let mut path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
        path.push("samples/test");
        train(path)
    }

    fn gen_test_header() -> ResponseHeader {
        let mut header = ResponseHeader::build(200, None).unwrap();
        header
            .append_header("Date", "Thu, 23 Dec 2021 11:23:29 GMT")
            .unwrap();
        header
            .append_header("Last-Modified", "Sat, 09 Oct 2021 22:41:34 GMT")
            .unwrap();
        header.append_header("Connection", "keep-alive").unwrap();
        header.append_header("Vary", "Accept-encoding").unwrap();
        header.append_header("Content-Encoding", "gzip").unwrap();
        header
            .append_header("Access-Control-Allow-Origin", "*")
            .unwrap();
        header
    }

    #[test]
    fn test_ser_with_dict() {
        let dict = gen_test_dict();
        let serde = crate::HeaderSerde::new(Some(dict));
        let serde_no_dict = crate::HeaderSerde::new(None);
        let header = gen_test_header();

        let compressed = serde.serialize(&header).unwrap();
        let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();
        let mut buf = vec![];
        let uncompressed = resp_header_to_buf(&header, &mut buf);

        assert!(compressed.len() < uncompressed);
        assert!(compressed.len() < compressed_no_dict.len());
    }

    #[test]
    fn test_deserialize_with_dict() {
        let dict = gen_test_dict();
        let serde = crate::HeaderSerde::new(Some(dict));
        let serde_no_dict = crate::HeaderSerde::new(None);
        let header = gen_test_header();

        let compressed = serde.serialize(&header).unwrap();
        let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();

        let from_dict_header = serde.deserialize(&compressed).unwrap();
        let from_no_dict_header = serde_no_dict.deserialize(&compressed_no_dict).unwrap();

        assert_eq!(from_dict_header.status, from_no_dict_header.status);
        assert_eq!(from_dict_header.headers, from_no_dict_header.headers);
    }

    #[test]
    fn test_ser_de_with_dict() {
        let dict = gen_test_dict();
        let serde = crate::HeaderSerde::new(Some(dict));
        let header = gen_test_header();

        let compressed = serde.serialize(&header).unwrap();
        let header2 = serde.deserialize(&compressed).unwrap();

        assert_eq!(header.status, header2.status);
        assert_eq!(header.headers, header2.headers);
    }
}

```

--------------------------------

### Test Empty Response Header Wire Format in Rust

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html?search=std%3A%3Avec

Tests the wire format of an empty response header. It builds a response header with a 200 status, serializes it to a byte buffer, and asserts the buffer's length and content match the expected HTTP/1.1 format. It also tests deserialization using `buf_to_http_header`.

```rust
#[test]
    fn test_empty_header_wire_format() {
        let header = ResponseHeader::build(200, None).unwrap();
        let mut buf = vec![];
        resp_header_to_buf(&header, &mut buf);

        // Should be: "HTTP/1.1 200 OK\r\n\r\n", total 19 bytes
        assert_eq!(buf.len(), 19);
        assert_eq!(buf, b"HTTP/1.1 200 OK\r\n\r\n");

        // Test that httparse can handle this
        let parsed = buf_to_http_header(&buf).unwrap();
        assert_eq!(parsed.status.as_u16(), 200);
    }
```

--------------------------------

### Serialize and Deserialize Response Headers in Rust

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html

Demonstrates the process of converting a ResponseHeader object into a raw byte buffer using 'resp_header_to_buf' and parsing it back using 'buf_to_http_header'. This verifies that the header structure is correctly maintained during wire transmission.

```rust
#[test]
fn test_empty_header_wire_format() {
    let header = ResponseHeader::build(200, None).unwrap();
    let mut buf = vec![];
    resp_header_to_buf(&header, &mut buf);

    assert_eq!(buf.len(), 19);
    assert_eq!(buf, b"HTTP/1.1 200 OK\r\n\r\n");

    let parsed = buf_to_http_header(&buf).unwrap();
    assert_eq!(parsed.status.as_u16(), 200);
}
```

--------------------------------

### Rust: Zstd Compression without Dictionary

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html

Provides functionality to compress data using Zstd without a pre-defined dictionary. It utilizes thread-local compression contexts for efficiency. The `compress` method takes raw data and a compression level, returning compressed data as a `Vec<u8>`. Dependencies include `zstd_safe` and `thread_local`.

```rust
use std::cell::{RefCell, RefMut};
use thread_local::ThreadLocal;
use zstd_safe::{CCtx, CDict, DCtx, DDict};

#[derive(Default)]
pub struct Compression(CompressionInner);

impl Compression {
    pub fn new() -> Self {
        Compression(CompressionInner::new())
    }

    pub fn compress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
        level: i32,
    ) -> Result<usize, &'static str> {
        self.0.compress_to_buffer(source, destination, level)
    }

    pub fn compress(&self, data: &[u8], level: i32) -> Result<Vec<u8>, &'static str> {
        let mut buffer = make_compressed_data_buffer(data.len());
        self.compress_to_buffer(data, &mut buffer, level)?;
        Ok(buffer)
    }

    pub fn decompress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
    ) -> Result<usize, &'static str> {
        self.0.decompress_to_buffer(source, destination)
    }
}

#[derive(Default)]
struct CompressionInner {
    com_context: ThreadLocal<RefCell<zstd_safe::CCtx<'static>>>,
    de_context: ThreadLocal<RefCell<zstd_safe::DCtx<'static>>>,
}

impl CompressionInner {
    fn new() -> Self {
        CompressionInner {
            com_context: ThreadLocal::new(),
            de_context: ThreadLocal::new(),
        }
    }

    #[inline]
    fn get_com_context(&self) -> RefMut<'_, CCtx<'static>> {
        self.com_context
            .get_or(|| RefCell::new(CCtx::create()))
            .borrow_mut()
    }

    #[inline]
    fn get_de_context(&self) -> RefMut<'_, DCtx<'static>> {
        self.de_context
            .get_or(|| RefCell::new(DCtx::create()))
            .borrow_mut()
    }

    fn compress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
        level: i32,
    ) -> Result<usize, &'static str> {
        self.get_com_context()
            .compress(destination, source, level)
            .map_err(zstd_safe::get_error_name)
    }

    fn decompress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
    ) -> Result<usize, &'static str> {
        self.get_de_context()
            .decompress(destination, source)
            .map_err(zstd_safe::get_error_name)
    }
}

// Placeholder for make_compressed_data_buffer, assuming it exists elsewhere
fn make_compressed_data_buffer(size: usize) -> Vec<u8> {
    Vec::with_capacity(size)
}

```

--------------------------------

### Type Mismatch in try_from Method Signature (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html?search=u32+-%3E+bool

Highlights a potential issue in the `pingora_header_serde::HeaderSerde::try_from` method where 'u32' is used as a generic parameter but not found, and 'U' is expected to match 'u32' while 'T' matches 'bool'. This indicates a possible type incompatibility or incorrect generic constraint.

```rust
method
pingora_header_serde::HeaderSerde::try_from
**U** -> Result<**T** >
where
u32 matches **U**
bool matches **T**
```

--------------------------------

### Test HTTP Header Serialization and Deserialization with Zstd Dictionary

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/dict.rs.html?search=

Tests the serialization and deserialization of HTTP response headers using `HeaderSerde`. It compares the compressed size with and without a zstd dictionary and verifies that deserialization reconstructs the original header correctly. This demonstrates the effectiveness of using a pre-trained dictionary for compression.

```rust
use super::*;
use crate::resp_header_to_buf;
use pingora_http::ResponseHeader;

fn gen_test_dict() -> Vec<u8> {
    let mut path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
    path.push("samples/test");
    train(path)
}

fn gen_test_header() -> ResponseHeader {
    let mut header = ResponseHeader::build(200, None).unwrap();
    header
        .append_header("Date", "Thu, 23 Dec 2021 11:23:29 GMT")
        .unwrap();
    header
        .append_header("Last-Modified", "Sat, 09 Oct 2021 22:41:34 GMT")
        .unwrap();
    header.append_header("Connection", "keep-alive").unwrap();
    header.append_header("Vary", "Accept-encoding").unwrap();
    header.append_header("Content-Encoding", "gzip").unwrap();
    header
        .append_header("Access-Control-Allow-Origin", "*")
        .unwrap();
    header
}

#[test]
fn test_ser_with_dict() {
    let dict = gen_test_dict();
    let serde = crate::HeaderSerde::new(Some(dict));
    let serde_no_dict = crate::HeaderSerde::new(None);
    let header = gen_test_header();

    let compressed = serde.serialize(&header).unwrap();
    let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();
    let mut buf = vec![];
    let uncompressed = resp_header_to_buf(&header, &mut buf);

    assert!(compressed.len() < uncompressed);
    assert!(compressed.len() < compressed_no_dict.len());
}

#[test]
fn test_deserialize_with_dict() {
    let dict = gen_test_dict();
    let serde = crate::HeaderSerde::new(Some(dict));
    let serde_no_dict = crate::HeaderSerde::new(None);
    let header = gen_test_header();

    let compressed = serde.serialize(&header).unwrap();
    let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();

    let from_dict_header = serde.deserialize(&compressed).unwrap();
    let from_no_dict_header = serde_no_dict.deserialize(&compressed_no_dict).unwrap();

    assert_eq!(from_dict_header.status, from_no_dict_header.status);
    assert_eq!(from_dict_header.headers, from_no_dict_header.headers);
}

#[test]
fn test_ser_de_with_dict() {
    let dict = gen_test_dict();
    let serde = crate::HeaderSerde::new(Some(dict));
    let header = gen_test_header();

    let compressed = serde.serialize(&header).unwrap();
    let header2 = serde.deserialize(&compressed).unwrap();

    assert_eq!(header.status, header2.status);
    assert_eq!(header.headers, header2.headers);
}
```

--------------------------------

### Train Zstd Dictionary from Files

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/dict.rs.html?search=

Trains a zstd dictionary from all files within a specified directory. It reads all files, concatenates their content, and generates a dictionary. This function is useful for compressing data more efficiently when the data characteristics are known beforehand.

```rust
use std::fs;
use zstd::dict;

/// Train the zstd dictionary from all the files under the given `dir_path`
///
/// The output will be the trained dictionary
pub fn train<P: AsRef<std::path::Path>>(dir_path: P) -> Vec<u8> {
    // TODO: check f is file, it can be dir
    let files = fs::read_dir(dir_path)
        .unwrap()
        .filter_map(|entry| entry.ok().map(|f| f.path()));
    dict::from_files(files, 64 * 1024 * 1024).unwrap()
}
```

--------------------------------

### Implement Thread-Local Zstandard Compression

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html?search=u32+-%3E+bool

Defines structures for standard and dictionary-based Zstandard compression. It uses 'CompressionInner' to manage thread-local 'CCtx' and 'DCtx' instances, ensuring efficient resource reuse across threads.

```rust
use std::cell::{RefCell, RefMut};
use thread_local::ThreadLocal;
use zstd_safe::{CCtx, CDict, DCtx, DDict};

#[derive(Default)]
pub struct Compression(CompressionInner);

impl Compression {
    pub fn compress(&self, data: &[u8], level: i32) -> Result<Vec<u8>, &'static str> {
        let mut buffer = make_compressed_data_buffer(data.len());
        self.compress_to_buffer(data, &mut buffer, level)?;
        Ok(buffer)
    }
}

#[derive(Default)]
struct CompressionInner {
    com_context: ThreadLocal<RefCell<zstd_safe::CCtx<'static>>>,
    de_context: ThreadLocal<RefCell<zstd_safe::DCtx<'static>>>,
}

impl CompressionInner {
    #[inline]
    fn get_com_context(&self) -> RefMut<'_, CCtx<'static>> {
        self.com_context
            .get_or(|| RefCell::new(CCtx::create()))
            .borrow_mut()
    }
}
```

--------------------------------

### Test empty HTTP response header serialization and parsing

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Verifies that an empty ResponseHeader is correctly serialized into the standard HTTP/1.1 wire format and can be successfully parsed back using buf_to_http_header. This test ensures the integrity of the header-serde implementation for minimal response objects.

```rust
#[test]
fn test_empty_header_wire_format() {
    let header = ResponseHeader::build(200, None).unwrap();
    let mut buf = vec![];
    resp_header_to_buf(&header, &mut buf);

    // Should be: "HTTP/1.1 200 OK\r\n\r\n", total 19 bytes
    assert_eq!(buf.len(), 19);
    assert_eq!(buf, b"HTTP/1.1 200 OK\r\n\r\n");

    // Test that httparse can handle this
    let parsed = buf_to_http_header(&buf).unwrap();
    assert_eq!(parsed.status.as_u16(), 200);
}
```

--------------------------------

### Implement From Trait - Rust

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=u32+-%3E+bool

Provides a basic `from` implementation where a type can be converted into itself. This is a standard blanket implementation.

```Rust
impl<T> From<T> for T
{
    fn from(t: T) -> T
}
```

--------------------------------

### HeaderSerde API

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html

This section details the `HeaderSerde` struct and its associated methods for serializing and deserializing HTTP response headers.

```APIDOC
## Struct HeaderSerde

### Description
HTTP Response header serialization. This struct provides the APIs to convert HTTP response header into compressed wired format for storage.

### Methods

#### `new(dict: Option<Vec<u8>>) -> Self`

**Description**: Create a new `HeaderSerde`. An optional zstd compression dictionary can be provided to improve the compression ratio and speed.

**Method**: `new` (constructor)

**Parameters**:
* `dict` (Option<Vec<u8>>) - Optional - A zstd compression dictionary.

#### `serialize(&self, header: &ResponseHeader) -> Result<Vec<u8>>`

**Description**: Serialize the given response header.

**Method**: `serialize`

**Parameters**:
* `header` (&ResponseHeader) - Required - The `ResponseHeader` to serialize.

**Returns**: `Result<Vec<u8>>` - A `Result` containing the serialized header data or an error.

#### `deserialize(&self, data: &[u8]) -> Result<ResponseHeader>`

**Description**: Deserialize the given response header.

**Method**: `deserialize`

**Parameters**:
* `data` (&[u8]) - Required - A slice of bytes representing the serialized header.

**Returns**: `Result<ResponseHeader>` - A `Result` containing the deserialized `ResponseHeader` or an error.
```

--------------------------------

### Deserialize HTTP Response Header with/without Dictionary (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/dict.rs.html

Tests the deserialization of `ResponseHeader` from compressed data, comparing results obtained using `HeaderSerde` with and without a pre-trained zstd dictionary. It verifies that headers serialized with or without a dictionary can be correctly deserialized, and that the resulting headers are identical.

```Rust
use super::*;
use crate::resp_header_to_buf;
use pingora_http::ResponseHeader;

fn gen_test_dict() -> Vec<u8> {
    let mut path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
    path.push("samples/test");
    train(path)
}

fn gen_test_header() -> ResponseHeader {
    let mut header = ResponseHeader::build(200, None).unwrap();
    header
        .append_header("Date", "Thu, 23 Dec 2021 11:23:29 GMT")
        .unwrap();
    header
        .append_header("Last-Modified", "Sat, 09 Oct 2021 22:41:34 GMT")
        .unwrap();
    header.append_header("Connection", "keep-alive").unwrap();
    header.append_header("Vary", "Accept-encoding").unwrap();
    header.append_header("Content-Encoding", "gzip").unwrap();
    header
        .append_header("Access-Control-Allow-Origin", "*")
        .unwrap();
    header
}

#[test]
fn test_deserialize_with_dict() {
    let dict = gen_test_dict();
    let serde = crate::HeaderSerde::new(Some(dict));
    let serde_no_dict = crate::HeaderSerde::new(None);
    let header = gen_test_header();

    let compressed = serde.serialize(&header).unwrap();
    let compressed_no_dict = serde_no_dict.serialize(&header).unwrap();

    let from_dict_header = serde.deserialize(&compressed).unwrap();
    let from_no_dict_header = serde_no_dict.deserialize(&compressed_no_dict).unwrap();

    assert_eq!(from_dict_header.status, from_no_dict_header.status);
    assert_eq!(from_dict_header.headers, from_no_dict_header.headers);
}
```

--------------------------------

### Header Serialization

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/struct.HeaderSerde.html?search=u32+-%3E+bool

Serializes a ResponseHeader object into a compressed byte vector.

```APIDOC
## Method: serialize

### Description
Converts a ResponseHeader into a compressed binary format suitable for storage.

### Parameters
#### Request Body
- **header** (ResponseHeader) - Required - The HTTP response header object to be serialized.

### Response
#### Success Response (200)
- **data** (Vec<u8>) - The serialized and compressed byte representation of the header.
```

--------------------------------

### Roundtrip Serialize and Deserialize HTTP Response Header (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/dict.rs.html

Tests the complete roundtrip process of serializing and then deserializing an HTTP `ResponseHeader` using `HeaderSerde` with a zstd dictionary. It ensures that the header object remains unchanged after being serialized and then deserialized, validating the integrity of the process.

```Rust
use super::*;
use crate::resp_header_to_buf;
use pingora_http::ResponseHeader;

fn gen_test_dict() -> Vec<u8> {
    let mut path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
    path.push("samples/test");
    train(path)
}

fn gen_test_header() -> ResponseHeader {
    let mut header = ResponseHeader::build(200, None).unwrap();
    header
        .append_header("Date", "Thu, 23 Dec 2021 11:23:29 GMT")
        .unwrap();
    header
        .append_header("Last-Modified", "Sat, 09 Oct 2021 22:41:34 GMT")
        .unwrap();
    header.append_header("Connection", "keep-alive").unwrap();
    header.append_header("Vary", "Accept-encoding").unwrap();
    header.append_header("Content-Encoding", "gzip").unwrap();
    header
        .append_header("Access-Control-Allow-Origin", "*")
        .unwrap();
    header
}

#[test]
fn test_ser_de_with_dict() {
    let dict = gen_test_dict();
    let serde = crate::HeaderSerde::new(Some(dict));
    let header = gen_test_header();

    let compressed = serde.serialize(&header).unwrap();
    let header2 = serde.deserialize(&compressed).unwrap();

    assert_eq!(header.status, header2.status);
    assert_eq!(header.headers, header2.headers);
}
```

--------------------------------

### Train zstd dictionary from directory in Rust

Source: https://docs.rs/pingora-header-serde/0.8.0/pingora_header_serde/dict/fn.train.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

The train function takes a path to a directory containing files and returns a byte vector representing the trained zstd dictionary. It requires the input path to implement the AsRef<Path> trait.

```rust
pub fn train<P: AsRef<Path>>(dir_path: P) -> Vec<u8>
```

--------------------------------

### Dictionary-Based Zstd Compression

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html?search=std%3A%3Avec

Provides a wrapper for Zstd compression using pre-loaded dictionaries. This is useful for compressing small messages where a shared dictionary can significantly improve the compression ratio.

```rust
pub struct CompressionWithDict {
    inner: CompressionInner,
    com_dict: CDict<'static>,
    de_dict: DDict<'static>,
}

impl CompressionWithDict {
    pub fn new(dict: &[u8], compression_level: i32) -> Self {
        CompressionWithDict {
            inner: CompressionInner::new(),
            com_dict: CDict::create(dict, compression_level),
            de_dict: DDict::create(dict),
        }
    }

    pub fn compress_to_buffer<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
    ) -> Result<usize, &'static str> {
        self.inner
            .compress_to_buffer_using_dict(source, destination, &self.com_dict)
    }
}
```

--------------------------------

### Create Compressed Data Buffer (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html?search=u32+-%3E+bool

Helper function to create a Vec<u8> buffer pre-allocated with sufficient capacity for compressed data. It calculates the maximum possible compressed size based on the uncompressed data length using zstd_safe::compress_bound.

```rust
fn make_compressed_data_buffer(uncompressed_len: usize) -> Vec<u8> {
    let buffer_len = zstd_safe::compress_bound(uncompressed_len);
    Vec::with_capacity(buffer_len)
}
```

--------------------------------

### Decompress Data Using Dictionary (Rust)

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/thread_zstd.rs.html

Decompresses source data from a byte slice into a destination buffer using a provided decompression dictionary (DDict). It returns the number of bytes written or an error string. This function requires a mutable WriteBuf for the destination and a DDict for decompression.

```Rust
pub fn decompress_to_buffer_using_dict<C: zstd_safe::WriteBuf + ?Sized>(
        &self,
        source: &[u8],
        destination: &mut C,
        dict: &DDict,
    ) -> Result<usize, &'static str> {
        self.get_de_context()
            .decompress_using_ddict(destination, source, dict)
            .map_err(zstd_safe::get_error_name)
    }
```

--------------------------------

### Test Response Header Serialization Without Dictionary

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html

Tests the serialization of a `ResponseHeader` without using a Zstandard dictionary. It asserts that the compressed output is smaller than the uncompressed buffer representation.

```rust
#[test]
fn test_ser_wo_dict() {
    let serde = HeaderSerde::new(None);
    let mut header = ResponseHeader::build(200, None).unwrap();
    header.append_header("foo", "bar").unwrap();
    header.append_header("foo", "barbar").unwrap();
    header.append_header("foo", "barbarbar").unwrap();
    header.append_header("Server", "Pingora").unwrap();

    let compressed = serde.serialize(&header).unwrap();
    let mut buf = vec![];
    let uncompressed = resp_header_to_buf(&header, &mut buf);
    assert!(compressed.len() < uncompressed);
}
```

--------------------------------

### Test Response Header Serialization With No Headers Added

Source: https://docs.rs/pingora-header-serde/0.8.0/src/pingora_header_serde/lib.rs.html

Tests the serialization of a `ResponseHeader` that has no custom headers added. It serializes and then deserializes the header to ensure the process works correctly even with an empty header set.

```rust
#[test]
fn test_no_headers() {
    let serde = HeaderSerde::new(None);
    let header = ResponseHeader::build(200, None).unwrap(); // No headers added

    // Serialize and deserialize
    let compressed = serde.serialize(&header).unwrap();

```
Tokens:
Raw