reqwest-ratelimit

https://github.com/xdarksome/reqwest-ratelimit
Admin
A rate-limit middleware implementation for reqwest-middleware that provides glue code for...
Tokens:1,707
Snippets:12
Trust Score:6.7
License:MIT
Update:8 months ago
Show doc for...
Tokens:
Raw
### Implement Custom RateLimiter Trait in Rust

Source: https://context7.com/xdarksome/reqwest-ratelimit/llms.txt

Demonstrates implementing the `RateLimiter` trait to create a custom token bucket rate limiter. This example shows how to manage tokens and acquire permits asynchronously, blocking until a permit is available. It uses atomic operations for thread-safe token management.

```rust
use std::future::Future;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use tokio::time::sleep;

// Custom rate limiter that allows N requests per second
struct TokenBucketLimiter {
    tokens: AtomicUsize,
    max_tokens: usize,
}

impl TokenBucketLimiter {
    fn new(max_tokens: usize) -> Self {
        Self {
            tokens: AtomicUsize::new(max_tokens),
            max_tokens,
        }
    }
}

impl reqwest_ratelimit::RateLimiter for TokenBucketLimiter {
    async fn acquire_permit(&self) {
        loop {
            let current = self.tokens.load(Ordering::SeqCst);
            if current > 0 {
                if self.tokens.compare_exchange(
                    current,
                    current - 1,
                    Ordering::SeqCst,
                    Ordering::SeqCst
                ).is_ok() {
                    return; // Permit acquired
                }
            }
            // Wait before retrying
            sleep(Duration::from_millis(100)).await;
        }
    }
}
```

--------------------------------

### Create Rate-Limited HTTP Client with reqwest-ratelimit in Rust

Source: https://context7.com/xdarksome/reqwest-ratelimit/llms.txt

Shows how to create a reqwest-middleware HTTP client with rate-limiting applied to all outgoing requests using the `all()` function. This example defines a simple `RateLimiter` implementation that introduces a delay between requests.

```rust
use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};

struct SimpleRateLimiter;

impl reqwest_ratelimit::RateLimiter for SimpleRateLimiter {
    async fn acquire_permit(&self) {
        // Implement rate limiting logic here
        // For example, sleep between requests:
        tokio::time::sleep(std::time::Duration::from_millis(100)).await;
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a rate-limited HTTP client
    let client = ClientBuilder::new(reqwest::Client::new())
        .with(reqwest_ratelimit::all(SimpleRateLimiter))
        .build();

    // All requests through this client will be rate-limited
    let response = client
        .get("https://api.example.com/data")
        .send()
        .await?;

    println!("Status: {}", response.status());
    Ok(())
}
```

--------------------------------

### Implement RateLimiter Trait and Use Middleware in Rust

Source: https://github.com/xdarksome/reqwest-ratelimit/blob/main/README.md

Demonstrates how to implement the `RateLimiter` trait and integrate it as middleware with `reqwest-middleware`. This involves defining a custom rate limiter struct and using `ClientBuilder` to attach the middleware. The example shows a basic `RateLimiter` that does nothing and how to send a request using the configured client.

```rust
use std::future::Future;
use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};

struct RateLimiter;

impl reqwest_ratelimit::RateLimiter for RateLimiter {
    async fn acquire_permit(&self) {
        // noop
    }
}

async fn run() {
    let client = ClientBuilder::new(reqwest::Client::new())
        .with(reqwest_ratelimit::all(RateLimiter))
        .build();

    client.get("https://crates.io").send().await.unwrap();
}
```

--------------------------------

### Implement Semaphore-Based RateLimiter for Concurrency in Rust

Source: https://context7.com/xdarksome/reqwest-ratelimit/llms.txt

Illustrates implementing the `RateLimiter` trait using `tokio::sync::Semaphore` to limit the number of concurrent requests. This approach ensures that no more than a specified number of requests are active simultaneously. Note that the current implementation drops the permit immediately, which might require adjustment for true concurrency limiting.

```rust
use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
use std::sync::Arc;
use tokio::sync::Semaphore;

// Rate limiter using a semaphore for concurrent request limiting
struct SemaphoreRateLimiter {
    semaphore: Arc<Semaphore>,
}

impl SemaphoreRateLimiter {
    fn new(max_concurrent: usize) -> Self {
        Self {
            semaphore: Arc::new(Semaphore::new(max_concurrent)),
        }
    }
}

impl reqwest_ratelimit::RateLimiter for SemaphoreRateLimiter {
    async fn acquire_permit(&self) {
        // This will block if max_concurrent permits are already acquired
        let _permit = self.semaphore.acquire().await.unwrap();
        // Note: permit is dropped immediately, for true concurrency limiting
        // you'd need a different approach
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let rate_limiter = SemaphoreRateLimiter::new(5); // Max 5 concurrent

    let client = ClientBuilder::new(reqwest::Client::new())
        .with(reqwest_ratelimit::all(rate_limiter))
        .build();

    // Make multiple requests - they will be rate-limited
    let urls = vec![
        "https://httpbin.org/get",
        "https://httpbin.org/ip",
        "https://httpbin.org/headers",
    ];

    for url in urls {
        let resp = client.get(url).send().await?;
        println!("{}: {}", url, resp.status());
    }

    Ok(())
}
```
Tokens:
Raw