### Complete Colly Example with Round-Robin Proxy Switching

Source: https://github.com/gocolly/colly/blob/master/_autodocs/proxy.md

A full example demonstrating how to initialize Colly, set up round-robin proxy switching, and define basic request, error, and HTML parsing logic.

```go
package main

import (
    "log"
    "github.com/gocolly/colly/v2"
    "github.com/gocolly/colly/v2/proxy"
)

func main() {
    c := colly.NewCollector()
    
    // Set up round-robin proxy switching
    proxySwitcher, err := proxy.RoundRobinProxySwitcher(
        "http://proxy1.example.com:8080",
        "http://proxy2.example.com:8080",
        "http://proxy3.example.com:8080",
    )
    if err != nil {
        log.Fatal(err)
    }
    c.SetProxyFunc(proxySwitcher)
    
    c.OnRequest(func(r *colly.Request) {
        log.Println("Visiting", r.URL)
    })
    
    c.OnError(func(_ *colly.Response, err error) {
        log.Println("Error:", err)
    })
    
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Attr("href")
        log.Println("Link:", link)
    })
    
    c.Visit("https://example.com")
}
```

--------------------------------

### Start Scraping with GET Request

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Initiates a scraping job by making a GET request to a specified URL. Returns an error if the request fails or is filtered.

```go
err := c.Visit("https://example.com")
if err != nil {
    log.Println(err)
}
```

--------------------------------

### Complete Colly Queue Example with Default Storage

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

A full example showing how to set up a Colly collector and a queue with default in-memory storage. It includes request logging, error handling, and duplicate URL tracking.

```go
package main

import (
    "log"
    "github.com/gocolly/colly/v2"
    "github.com/gocolly/colly/v2/queue"
)

func main() {
    // Create queue with 4 threads and default storage
    q, _ := queue.New(4, nil)
    
    c := colly.NewCollector()
    
    // Track visited URLs
    visited := make(map[string]bool)
    
    c.OnRequest(func(r *colly.Request) {
        log.Println("Visiting", r.URL)
    })
    
    c.OnError(func(_ *colly.Response, err error) {
        log.Println("Error:", err)
    })
    
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Request.AbsoluteURL(e.Attr("href"))
        
        if !visited[link] {
            visited[link] = true
            q.AddURL(link)
        }
    })
    
    // Start with seed URL
    q.AddURL("https://example.com")
    
    // Run queue with 4 concurrent requests
    q.Run(c)
}
```

--------------------------------

### Run Rate Limiting Example

Source: https://github.com/gocolly/colly/blob/master/_examples/README.md

Demonstrates how to use rate limiting in Colly. Execute this example using `go run rate_limit/rate_limit.go`.

```bash
$ go run rate_limit/rate_limit.go
[000001] 1 [     1 - request] map["url":"https://httpbin.org/delay/2?n=4"] (60.872µs)
[000002] 1 [     2 - request] map["url":"https://httpbin.org/delay/2?n=2"] (154.425µs)
[000003] 1 [     3 - request] map["url":"https://httpbin.org/delay/2?n=0"] (158.374µs)
[000004] 1 [     5 - request] map["url":"https://httpbin.org/delay/2?n=3"] (426.999µs)
[000005] 1 [     4 - request] map["url":"https://httpbin.org/delay/2?n=1"] (448.75µs)
[000007] 1 [     2 - response] map["url":"https://httpbin.org/delay/2?n=2" "status":"OK"] (2.855764394s)
[000008] 1 [     2 - scraped] map["url":"https://httpbin.org/delay/2?n=2"] (2.855797868s)
[000006] 1 [     1 - response] map["url":"https://httpbin.org/delay/2?n=4" "status":"OK"] (2.855756753s)
[000009] 1 [     1 - scraped] map["url":"https://httpbin.org/delay/2?n=4"] (2.855819581s)
[000010] 1 [     3 - response] map["status":"OK" "url":"https://httpbin.org/delay/2?n=0"] (5.002065299s)
[000011] 1 [     3 - scraped] map["url":"https://httpbin.org/delay/2?n=0"] (5.002103755s)
[000012] 1 [     5 - response] map["status":"OK" "url":"https://httpbin.org/delay/2?n=3"] (5.012080614s)
[000013] 1 [     5 - scraped] map["url":"https://httpbin.org/delay/2?n=3"] (5.012101056s)
[000014] 1 [     4 - response] map["url":"https://httpbin.org/delay/2?n=1" "status":"OK"] (7.155725591s)
[000015] 1 [     4 - scraped] map["url":"https://httpbin.org/delay/2?n=1"] (7.155759136s)

```

--------------------------------

### DomainRegexp Examples

Source: https://github.com/gocolly/colly/blob/master/_autodocs/rate-limiting.md

Illustrates various regular expression patterns for matching domains. Use these for precise domain filtering.

```go
// Match exactly example.com
DomainRegexp: "^example\\.com$"
```

```go
// Match example.com and subdomains
DomainRegexp: "(.+\\.)?example\\.com"
```

```go
// Match any api domain
DomainRegexp: "api\\..+"
```

```go
// Match numeric subdomains
DomainRegexp: "server-[0-9]+\\.example\\.com"
```

--------------------------------

### Basic Web Crawler in Go

Source: https://github.com/gocolly/colly/blob/master/README.md

This snippet demonstrates how to create a basic web crawler using GoColly. It visits a starting URL and recursively visits all links found on the page. Ensure you have the GoColly library installed.

```go
import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}
```

--------------------------------

### DomainGlob Examples

Source: https://github.com/gocolly/colly/blob/master/_autodocs/rate-limiting.md

Demonstrates glob patterns for simple domain matching. Useful for broader domain filtering.

```go
// Match exactly example.com
DomainGlob: "example.com"
```

```go
// Match subdomains
DomainGlob: "*.example.com"
```

```go
// Match multiple levels
DomainGlob: "*.api.*.example.com"
```

```go
// Match numbered servers
DomainGlob: "api-*.example.com"
```

--------------------------------

### Visit

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Starts a scraping job by making a GET request to the specified URL. It can return various errors if the request fails or is filtered.

```APIDOC
## Visit

### Description
Starts a scraping job by making a GET request to the specified URL.

### Method
GET

### Endpoint
[URL]

### Parameters
#### Path Parameters
- **URL** (string) - Required - The URL to visit

### Returns
- **error** - Returns error if the request fails or is filtered. May return various error types: ErrForbiddenDomain, ErrForbiddenURL, ErrMaxDepth, ErrRobotsTxtBlocked, etc.

### Example
```go
err := c.Visit("https://example.com")
if err != nil {
    log.Println(err)
}
```
```

--------------------------------

### ErrQueueFull Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Demonstrates handling the ErrQueueFull error when the queue's maximum size is reached. Ensure the queue is initialized with a MaxSize.

```go
var ErrQueueFull = errors.New("Queue MaxSize reached")
```

```go
q, _ := queue.New(2, &queue.InMemoryQueueStorage{MaxSize: 100})
for i := 0; i < 101; i++ {
    err := q.AddURL("https://example.com/" + string(rune(i)))
    if err == colly.ErrQueueFull {
        log.Println("Queue capacity exceeded")
    }
}
```

--------------------------------

### Visit Method for GET Requests

Source: https://github.com/gocolly/colly/blob/master/_autodocs/request.md

Continues a collector job by making a GET request. Preserves the context of the original request and accepts a URL string for the next visit.

```go
c.OnResponse(func(r *colly.Response) {
    r.Request.Visit(r.Request.URL.String() + "/page2")
})
```

--------------------------------

### Start Scraping with POST Request (Multipart)

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Initiates a collector job by creating a Multipart POST request with binary data. Returns an error if the request fails.

```go
func (c *Collector) PostMultipart(URL string, requestData map[string][]byte) error
```

--------------------------------

### Proxy with Rate Limiting Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/proxy.md

Demonstrates how to combine a round-robin proxy selector with rate limiting rules for a specific domain. This ensures controlled scraping of target websites.

```go
import (
    "time"
    "net/http"
    "github.com/gocolly/colly"
    "github.com/gocolly/colly/proxy"
)

c := colly.NewCollector(
    colly.AllowedDomains("example.com"),
    colly.MaxDepth(2),
)

// Set round-robin proxy
proxySwitcher, _ := proxy.RoundRobinProxySwitcher(
    "http://proxy1:8080",
    "http://proxy2:8080",
)
c.SetProxyFunc(proxySwitcher)

// Add rate limiting
c.Limit(&colly.LimitRule{
    DomainRegexp: "example\.com",
    Delay:        1 * time.Second,
    Parallelism:  2,
})

c.Visit("https://example.com")
```

--------------------------------

### Start Scraping with POST Request (Raw Data)

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Initiates a collector job by creating a POST request with raw binary data. Returns an error if the request fails.

```go
func (c *Collector) PostRaw(URL string, requestData []byte) error
```

--------------------------------

### Following Links with Colly

Source: https://github.com/gocolly/colly/blob/master/_autodocs/README.md

This example shows how to follow links found on a page. It restricts scraping to a specific domain and visits discovered URLs.

```go
c := colly.NewCollector(colly.AllowedDomains("example.com"))
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    link := e.Request.AbsoluteURL(e.Attr("href"))
    e.Request.Visit(link)
})
c.Visit("https://example.com")
```

--------------------------------

### ErrEmptyProxyURL Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Returned when RoundRobinProxySwitcher is invoked with an empty list of proxy URLs. Always provide a non-empty list of proxies to this function.

```go
var ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
```

```go
_, err := proxy.RoundRobinProxySwitcher() // No URLs provided
// err == colly.ErrEmptyProxyURL
```

--------------------------------

### Start Scraping with POST Request (Form Data)

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Initiates a collector job by creating a POST request with form data. Returns an error if the request fails.

```go
err := c.Post("https://example.com/login", map[string]string{
    "username": "user",
    "password": "pass",
})
```

--------------------------------

### ErrMaxRequests Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Shows how to handle ErrMaxRequests when the collector exceeds the configured maximum number of requests. Initialize the collector with colly.MaxRequests.

```go
var ErrMaxRequests = errors.New("Max Requests limit reached")
```

```go
c := colly.NewCollector(colly.MaxRequests(10))
// After 10 requests:
err := c.Visit("https://example.com/11")
// err == colly.ErrMaxRequests
```

--------------------------------

### Process URLs with a Queue

Source: https://github.com/gocolly/colly/blob/master/_autodocs/README.md

Manage a queue of URLs to be scraped, allowing for concurrent processing. This example initializes a queue with a concurrency limit and adds an initial URL.

```go
q, _ := queue.New(4, nil)
q.AddURL("https://example.com/1")
q.Run(c)
```

--------------------------------

### Start Scraping with HEAD Request

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Initiates a collector job by creating a HEAD request to a specified URL. Returns an error if the request fails.

```go
func (c *Collector) Head(URL string) error
```

--------------------------------

### Environment Variable Configuration Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/configuration.md

Set Colly configuration options using environment variables before initializing the collector in Go. This allows for flexible configuration without code changes.

```bash
export COLLY_USER_AGENT="MyBot/1.0"
export COLLY_ALLOWED_DOMAINS="example.com,api.example.com"
export COLLY_MAX_DEPTH="5"
export COLLY_MAX_REQUESTS="10000"
export COLLY_CACHE_DIR="/tmp/colly_cache"
export COLLY_IGNORE_ROBOTSTXT="yes"
export COLLY_TRACE_HTTP="yes"
export COLLY_DETECT_CHARSET="yes"
```

```go
// Reads environment variables
c := colly.NewCollector()
// c.UserAgent = "MyBot/1.0"
// c.AllowedDomains = ["example.com", "api.example.com"]
// c.MaxDepth = 5
```

--------------------------------

### Check for ErrQueueFull

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Example of how to check if an error returned from AddURL is ErrQueueFull, indicating the queue is full and the URL should be skipped.

```go
err := q.AddURL(url)
if err == colly.ErrQueueFull {
    log.Println("Queue is full, skipping URL")
}
```

--------------------------------

### Post

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Starts a collector job by creating a POST request with form data. Returns an error if the request fails.

```APIDOC
## Post

### Description
Starts a collector job by creating a POST request with form data.

### Method
POST

### Endpoint
[URL]

### Parameters
#### Path Parameters
- **URL** (string) - Required - The URL for the POST request
#### Request Body
- **requestData** (map[string]string) - Required - Form data to send

### Returns
- **error** - Returns error if the request fails.

### Example
```go
err := c.Post("https://example.com/login", map[string]string{
    "username": "user",
    "password": "pass",
})
```
```

--------------------------------

### Basic HTML Scraping with Colly

Source: https://github.com/gocolly/colly/blob/master/_autodocs/README.md

A simple example demonstrating how to scrape data from HTML elements. It targets `a` tags with an `href` attribute and prints the URL.

```go
c := colly.NewCollector()
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    fmt.Println(e.Attr("href"))
})
c.Visit("https://example.com")
```

--------------------------------

### Handle ErrEmptyProxyURL Error

Source: https://github.com/gocolly/colly/blob/master/_autodocs/proxy.md

Example of how to check for and handle the ErrEmptyProxyURL error when calling RoundRobinProxySwitcher.

```go
_, err := proxy.RoundRobinProxySwitcher() // Error
if err == colly.ErrEmptyProxyURL {
    log.Println("Must provide at least one proxy URL")
}
```

--------------------------------

### PostRaw

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Starts a collector job by creating a POST request with raw binary data. Returns an error if the request fails.

```APIDOC
## PostRaw

### Description
Starts a collector job by creating a POST request with raw binary data.

### Method
POST

### Endpoint
[URL]

### Parameters
#### Path Parameters
- **URL** (string) - Required - The URL for the POST request
#### Request Body
- **requestData** ([]byte) - Required - Raw binary data to send

### Returns
- **error** - Returns error if the request fails.
```

--------------------------------

### Apply Multiple Extensions to Colly Collector

Source: https://github.com/gocolly/colly/blob/master/_autodocs/extensions.md

Demonstrates how to initialize a Colly collector with asynchronous mode and allowed domains, then apply multiple extensions like RandomUserAgent. It also shows how to set request timeouts and rate limits before starting the scraping process.

```go
c := colly.NewCollector(
    colly.Async(),
    colly.AllowedDomains("example.com"),
)

// Apply multiple extensions
extensions.RandomUserAgent(c)

// May add referer and other extensions here
// extensions.Referer(c) // if available

c.SetRequestTimeout(30 * time.Second)
c.Limit(&colly.LimitRule{
    DomainGlob: "example.com",
    Delay:      2 * time.Second,
})

c.Visit("https://example.com")
c.Wait()
```

--------------------------------

### Visit

Source: https://github.com/gocolly/colly/blob/master/_autodocs/request.md

Continues the collector job by making a GET request, preserving the context of the original request.

```APIDOC
## Visit

### Description
Continues the collector job by making a GET request, preserving the context of the original request.

### Method
GET

### Endpoint
[URL]

### Parameters
#### Query Parameters
- **URL** (string) - Required - URL to visit (relative or absolute)

### Returns
- error - Error if request fails or is filtered.

### Example
```go
c.OnResponse(func(r *colly.Response) {
    r.Request.Visit(r.Request.URL.String() + "/page2")
})
```
```

--------------------------------

### Configure Cache Directory

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Specifies a directory where GET requests are cached as files. Caching is disabled by default.

```go
c := colly.NewCollector(colly.CacheDir("./cache"))
```

--------------------------------

### Head

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Starts a collector job by creating a HEAD request to the specified URL. Returns an error if the request fails.

```APIDOC
## Head

### Description
Starts a collector job by creating a HEAD request.

### Method
HEAD

### Endpoint
[URL]

### Parameters
#### Path Parameters
- **URL** (string) - Required - The URL for the HEAD request

### Returns
- **error** - Returns error if the request fails.
```

--------------------------------

### Integer Parsing in Environment Variables

Source: https://github.com/gocolly/colly/blob/master/_autodocs/configuration.md

Provides examples of environment variables for integer-based configurations, such as maximum depth, request count, and body size. Values are expected as decimal integers.

```bash
COLLY_MAX_DEPTH="10"
COLLY_MAX_REQUESTS="5000"
COLLY_MAX_BODY_SIZE="52428800"  # 50MB
```

--------------------------------

### Get

Source: https://github.com/gocolly/colly/blob/master/_autodocs/context.md

Retrieves a string value from the context using its key. If the key is not found or the value is not a string, an empty string is returned.

```APIDOC
## Get

### Description
Retrieves a string value from the context. Returns empty string if key not found or value is not a string.

### Parameters
#### Path Parameters
- **key** (string) - Required - Key to retrieve

### Returns
string - Value as string or empty string.

### Example
```go
c.OnRequest(func(r *colly.Request) {
    r.Ctx.Put("user_id", "123")
})

c.OnResponse(func(r *colly.Response) {
    userID := r.Ctx.Get("user_id")
    log.Println("User:", userID)
})
```
```

--------------------------------

### Start Scraping with Custom HTTP Request

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Initiates a collector job with a custom HTTP request, allowing full control over method, headers, and body. Returns an error if the request fails.

```go
func (c *Collector) Request(method, URL string, requestData io.Reader, ctx *Context, hdr http.Header) error
```

--------------------------------

### ErrForbiddenDomain Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Returned when visiting a domain not in AllowedDomains or present in DisallowedDomains. Use this to catch and handle attempts to scrape unauthorized domains.

```go
var ErrForbiddenDomain = errors.New("Forbidden domain")
```

```go
c := colly.NewCollector(colly.AllowedDomains("example.com"))
err := c.Visit("https://other.com")
// err == colly.ErrForbiddenDomain
```

```go
c.OnError(func(r *colly.Response, err error) {
    if err == colly.ErrForbiddenDomain {
        log.Println("Domain not allowed")
    }
})
```

--------------------------------

### PostMultipart

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Starts a collector job by creating a Multipart POST request with binary data. Returns an error if the request fails.

```APIDOC
## PostMultipart

### Description
Starts a collector job by creating a Multipart POST request with binary data.

### Method
POST

### Endpoint
[URL]

### Parameters
#### Path Parameters
- **URL** (string) - Required - The URL for the POST request
#### Request Body
- **requestData** (map[string][]byte) - Required - Multipart form data

### Returns
- **error** - Returns error if the request fails.
```

--------------------------------

### Domain List Parsing in Environment Variables

Source: https://github.com/gocolly/colly/blob/master/_autodocs/configuration.md

Illustrates how comma-separated domains are parsed from environment variables. The example shows a typical format for restricting crawling to specific domains.

```bash
COLLY_ALLOWED_DOMAINS="example.com,api.example.com,cdn.example.com"
# Parsed as: []string{"example.com", "api.example.com", "cdn.example.com"}
```

--------------------------------

### AlreadyVisitedError Handling Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Demonstrates how to detect and handle an AlreadyVisitedError when visiting a URL that has already been processed and AllowURLRevisit is false. Type assertion is used to check the error type and access the destination URL.

```go
c := colly.NewCollector() // AllowURLRevisit defaults to false

c.Visit("https://example.com")
err := c.Visit("https://example.com") // Already visited

if avErr, ok := err.(*colly.AlreadyVisitedError); ok {
    log.Printf("Already visited: %s\n", avErr.Destination)
    log.Println(avErr.Error()) // Output: "https://example.com" already visited
}
```

--------------------------------

### PostgreSQL Storage for Distributed Crawling

Source: https://github.com/gocolly/colly/blob/master/_autodocs/storage.md

Example of a custom storage implementation using PostgreSQL to track visited URLs across multiple crawler instances. This ensures that each URL is processed only once.

```go
// Multiple processes with shared PostgreSQL storage
type PostgresStorage struct {
    db *sql.DB
}

func (ps *PostgresStorage) Visited(id uint64) error {
    _, err := ps.db.Exec("INSERT INTO visited (hash) VALUES ($1)", id)
    return err
}
```

--------------------------------

### Request

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Starts a collector job with a custom HTTP request, allowing full control over method, headers, and body. Returns an error if the request fails.

```APIDOC
## Request

### Description
Starts a collector job with a custom HTTP request. Allows full control over method, headers, and body.

### Method
[method]

### Endpoint
[URL]

### Parameters
#### Path Parameters
- **method** (string) - Required - HTTP method (GET, POST, PUT, DELETE, PATCH, OPTIONS, HEAD)
- **URL** (string) - Required - The request URL
- **requestData** (io.Reader) - Optional - Request body (nil for no body)
- **ctx** (*Context) - Optional - Context for passing data between callbacks (nil to create new)
- **hdr** (http.Header) - Optional - Custom HTTP headers (nil to use defaults)

### Returns
- **error** - Returns error if the request fails.
```

--------------------------------

### Run

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Starts the queue consumer threads and processes all requests using the provided collector. This method blocks until the queue is empty and all requests are complete.

```APIDOC
## Run

### Description
Starts the queue consumer threads and processes all requests using the provided collector. Blocks until queue is empty and all requests complete.

### Method
`Run`

### Parameters
#### Path Parameters
- **c** (*colly.Collector) - Required - Collector to execute requests

### Response
#### Success Response (error)
- **error** - Storage or collection errors.

### Request Example
```go
c := colly.NewCollector()
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    link := e.Attr("href")
    q.AddURL(e.Request.AbsoluteURL(link))
})

q.Run(c)
```
```

--------------------------------

### ErrNoPattern Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Occurs when LimitRule.Init() is called without specifying either DomainRegexp or DomainGlob. Ensure at least one domain matching pattern is provided for rate limiting rules.

```go
var ErrNoPattern = errors.New("No pattern defined in LimitRule")
```

```go
rule := &colly.LimitRule{
    Delay: 1 * time.Second,
    // Neither DomainRegexp nor DomainGlob specified
}
err := c.Limit(rule)
// err == colly.ErrNoPattern
```

--------------------------------

### Implement Custom Proxy Logic (Go)

Source: https://github.com/gocolly/colly/blob/master/_autodocs/types.md

Example of setting a custom proxy function for a Colly collector. This function parses the request URL to decide which proxy to use.

```go
c.SetProxyFunc(func(req *http.Request) (*url.URL, error) {
    if strings.Contains(req.URL.Host, "api") {
        return url.Parse("http://api-proxy:8080")
    }
    return url.Parse("http://default-proxy:8080")
})
```

--------------------------------

### ErrRobotsTxtBlocked Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Returned when a URL is disallowed by the website's robots.txt file and IgnoreRobotsTxt is set to false. Be mindful of robots.txt rules to avoid being blocked.

```go
var ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
```

```go
// robots.txt contains: User-agent: * / Disallow: /private/
c := colly.NewCollector() // IgnoreRobotsTxt defaults to true
err := c.Visit("https://example.com/private/")
// May return ErrRobotsTxtBlocked if IgnoreRobotsTxt is disabled
```

--------------------------------

### Log HTTP Request Latency (Go)

Source: https://github.com/gocolly/colly/blob/master/_autodocs/types.md

Example of how to access and log HTTP request timing information using the TraceHTTP collector option. It calculates and prints connection and data transfer durations.

```go
c := colly.NewCollector(colly.TraceHTTP())
c.OnResponse(func(r *colly.Response) {
    if r.Trace != nil {
        log.Printf("Latency: %v + %v\n",
            r.Trace.ConnectDuration,
            r.Trace.FirstByteDuration - r.Trace.ConnectDuration,
        )
    }
})
```

--------------------------------

### Using Default InMemoryStorage

Source: https://github.com/gocolly/colly/blob/master/_autodocs/storage.md

Demonstrates how to create a new collector, which uses InMemoryStorage by default. It also shows how to explicitly set an InMemoryStorage instance.

```go
c := colly.NewCollector()
// InMemoryStorage is set by default

// Or explicitly set it
storage := &storage.InMemoryStorage{}
c.SetStorage(storage)
```

--------------------------------

### Apply Rate Limiting Rule (Go)

Source: https://github.com/gocolly/colly/blob/master/_autodocs/types.md

Example of creating and applying a LimitRule to a Colly collector. This rule specifies a glob pattern for domains, a fixed delay, a random delay, and the maximum number of parallel requests.

```go
rule := &colly.LimitRule{
    DomainGlob:  "*.example.com",
    Delay:       1 * time.Second,
    RandomDelay: 500 * time.Millisecond,
    Parallelism: 2,
}
c.Limit(rule)
```

--------------------------------

### Parse XML with XPath

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Extract data from XML content using XPath selectors. Use `ChildText` to get text content and `ChildAttr` to get attribute values.

```go
c.OnXML("//item", func(e *colly.XMLElement) {
    title := e.ChildText("title")
    author := e.ChildAttr("author", "name")
})
```

--------------------------------

### CacheDir

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Specifies a directory where GET requests are cached as files.

```APIDOC
## CacheDir

### Description
Specifies a directory where GET requests are cached as files.

### Signature
```go
func CacheDir(path string) CollectorOption
```

### Parameters
#### Path
- **path** (string) - Cache directory path. Default: "" (disabled)

### Example
```go
c := colly.NewCollector(colly.CacheDir("./cache"))
```
```

--------------------------------

### CheckHead

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Performs a HEAD request before every GET to pre-validate the response.

```APIDOC
## CheckHead

### Description
Performs a HEAD request before every GET to pre-validate the response.

### Method
CollectorOption

### Endpoint
N/A

### Parameters
None

### Request Example
N/A

### Response
N/A
```

--------------------------------

### Apply Multiple Collector Options (Go)

Source: https://github.com/gocolly/colly/blob/master/_autodocs/types.md

Demonstrates how to apply multiple configuration options when creating a new Colly collector. This includes setting the User-Agent, maximum depth, and allowed domains.

```go
c := colly.NewCollector(
    colly.UserAgent("Bot/1.0"),
    colly.MaxDepth(3),
    colly.AllowedDomains("example.com"),
)
```

--------------------------------

### Perform HEAD Requests in Go

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Use the Head method to send a HEAD request to a URL. Alternatively, configure the collector with CheckHead() to perform a HEAD request before a GET request for validation.

```go
c.Head("https://example.com")

// or with validation before GET
c := colly.NewCollector(colly.CheckHead())
c.Visit("https://example.com")
```

--------------------------------

### Initialize New Context

Source: https://github.com/gocolly/colly/blob/master/_autodocs/context.md

Creates a new, empty Context instance. Contexts are automatically generated per request, but manual creation is also supported.

```go
func NewContext() *Context
```

--------------------------------

### IsAbort

Source: https://github.com/gocolly/colly/blob/master/_autodocs/request.md

Checks if the request has been aborted. This is useful for determining if a request was intentionally cancelled, for example, in an OnRequest callback.

```APIDOC
## IsAbort

### Description
Returns true if the request has been aborted via Abort().

### Returns
- **bool**: Abort state.

### Source
request.go:94
```

--------------------------------

### Using InMemoryQueueStorage with a Custom MaxSize

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Demonstrates initializing InMemoryQueueStorage with a specific maximum capacity and adding URLs to the queue. This is useful for limiting memory usage.

```go
storage := &queue.InMemoryQueueStorage{
    MaxSize: 100000,
}
q, _ := queue.New(4, storage)

q.AddURL("https://example.com/1")
q.AddURL("https://example.com/2")

c := colly.NewCollector()
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    q.AddURL(e.Request.AbsoluteURL(e.Attr("href")))
})

_ = q.Run(c)
```

--------------------------------

### Create and Assign Context

Source: https://github.com/gocolly/colly/blob/master/_autodocs/context.md

Demonstrates how to manually create a new Context and assign it to a request's Ctx field.

```go
ctx := colly.NewContext()
r.Ctx = ctx
```

--------------------------------

### Rate Limiting

Source: https://github.com/gocolly/colly/blob/master/_autodocs/MANIFEST.txt

Controls the rate of requests made to a specific domain to avoid overwhelming servers or getting blocked.

```APIDOC
## Rate Limiting

### Description
Controls the rate of requests per domain.

### Types
- `LimitRule`: Defines rate limiting parameters.
  - `DomainGlobs` ([]string): List of domain patterns to apply the rule to.
  - `RandomDelay` (time.Duration): Random delay to add to each request.
  - `Parallelism` (int): Maximum number of concurrent requests per domain.

### Usage
Use the `Limit()` method on a Collector instance with a `LimitRule`.

### Example
```go
collector.Limit(&colly.LimitRule{DomainGlobs: []string{"*.example.com"}, Parallelism: 2, RandomDelay: 5 * time.Second})
```
```

--------------------------------

### Visit URL with Context

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Create a new request with a custom context, allowing you to pass additional data like user IDs. The `Do` method then executes the request.

```go
req, _ := c.NewCollector().Request("GET", url, nil, nil, nil)
req.Ctx.Put("user_id", "123")
req.Do()
```

--------------------------------

### ErrNoCookieJar Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Triggered if SetCookies is called after DisableCookies() has been invoked. This error prevents cookie manipulation when cookies are explicitly disabled.

```go
var ErrNoCookieJar = errors.New("Cookie jar is not available")
```

```go
c := colly.NewCollector()
c.DisableCookies()
err := c.SetCookies("https://example.com", cookies)
// err == colly.ErrNoCookieJar
```

--------------------------------

### Override Environment Variables with Constructor Options

Source: https://github.com/gocolly/colly/blob/master/_autodocs/configuration.md

Demonstrates how explicit constructor options take precedence over environment variables for configuration. The UserAgent is set via both methods, with the constructor option being the effective one.

```go
os.Setenv("COLLY_USER_AGENT", "EnvBot")
c := colly.NewCollector(colly.UserAgent("OptionBot"))
// c.UserAgent = "OptionBot" (option overrides env)
```

--------------------------------

### ErrForbiddenURL Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Returned when a URL matches a pattern defined in DisallowedURLFilters. Use this to prevent scraping of specific URL paths.

```go
var ErrForbiddenURL = errors.New("ForbiddenURL")
```

```go
c := colly.NewCollector(
    colly.DisallowedURLFilters(regexp.MustCompile("admin")),
)
err := c.Visit("https://example.com/admin")
// err == colly.ErrForbiddenURL
```

--------------------------------

### Configuration Options

Source: https://github.com/gocolly/colly/blob/master/_autodocs/MANIFEST.txt

Lists all available constructor options and environment variables for configuring the Colly collector.

```APIDOC
## Configuration

### Description
Provides details on constructor options and environment variables.

### Constructor Options
- `MaxDepth` (int): Maximum depth of the crawl.
- `UserAgent` (string): User-Agent string to use.
- `AllowedDomains` ([]string): List of domains allowed for crawling.
- `IgnoreDomains` ([]string): List of domains to ignore during crawling.
- `URLFilters` ([]*regexp.Regexp): Regular expressions to filter URLs.
- `AsyncEnabled` (bool): Enables asynchronous request processing.
- `CacheDir` (string): Directory for caching responses.
- `RequestTimeout` (time.Duration): Timeout for individual requests.
- `MaxBodySize` (int): Maximum response body size to process.
- `Transport` (http.RoundTripper): Custom HTTP transport.
- `ParseHTTPResponse` (bool): Whether to parse the HTTP response body.
- `ID` (string): Unique identifier for the collector.
- `Collector` (*Collector): Embed another collector's configuration.

### Environment Variables
- `HTTP_PROXY`: Sets the HTTP proxy.
- `HTTPS_PROXY`: Sets the HTTPS proxy.
- `NO_PROXY`: Comma-separated list of hosts to bypass proxy.

### Usage
Pass these options as a struct to the `colly.NewCollector()` function.
```

--------------------------------

### ErrMaxDepth Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Triggered when the maximum allowed scraping depth is exceeded. Configure MaxDepth to control how deep Colly should crawl.

```go
var ErrMaxDepth = errors.New("Max depth limit reached")
```

```go
c := colly.NewCollector(colly.MaxDepth(2))
// First Visit: depth 1 - OK
// OnHTML Visit: depth 2 - OK
// OnHTML Visit: depth 3 - ErrMaxDepth
```

--------------------------------

### Create New Collector Instance

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Creates a new Collector instance. Use options to configure domains, user agent, and depth.

```go
c := colly.NewCollector()
c := colly.NewCollector(
    colly.AllowedDomains("example.com"),
    colly.UserAgent("My Scraper"),
    colly.MaxDepth(2),
)
```

--------------------------------

### Retrieve Cookies for a URL

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Get the cookies that would be sent to a specific URL. Returns nil if the cookie jar is disabled.

```go
func (c *Collector) Cookies(URL string) []*http.Cookie
```

--------------------------------

### Create New Request with New

Source: https://github.com/gocolly/colly/blob/master/_autodocs/request.md

Use `New` to create a new request instance, inheriting the context of the original request. This is ideal for building complex or dynamic requests programmatically.

```go
newReq, _ := r.New("PUT", "https://example.com/api", bytes.NewReader(data))
newReq.Do()
```

--------------------------------

### Add URL to Queue

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Adds a new URL to the queue for processing as a GET request. Returns an error if the queue is full.

```go
q.AddURL("https://example.com/page1")
q.AddURL("https://example.com/page2")
```

--------------------------------

### InMemoryStorage Usage

Source: https://github.com/gocolly/colly/blob/master/_autodocs/storage.md

Demonstrates how to use the default InMemoryStorage, either by default or by explicitly setting it.

```APIDOC
## InMemoryStorage

### Usage

```go
c := colly.NewCollector()
// InMemoryStorage is set by default

// Or explicitly set it
storage := &storage.InMemoryStorage{}
c.SetStorage(storage)
```
```

--------------------------------

### Custom RedisStorage Implementation

Source: https://github.com/gocolly/colly/blob/master/_autodocs/storage.md

Provides a concrete implementation of the Storage interface using Redis for persistence. This includes methods for initializing the connection, tracking visited URLs, and managing cookies.

```go
type RedisStorage struct {
    client *redis.Client
}

func (r *RedisStorage) Init() error {
    // Initialize Redis connection
    return nil
}

func (r *RedisStorage) Visited(requestID uint64) error {
    // Store visited URL hash in Redis
    return r.client.Set(ctx, fmt.Sprintf("visited:%d", requestID), true, 0).Err()
}

func (r *RedisStorage) IsVisited(requestID uint64) (bool, error) {
    // Check if URL hash exists in Redis
    exists := r.client.Exists(ctx, fmt.Sprintf("visited:%d", requestID))
    return exists.Val() > 0, exists.Err()
}

func (r *RedisStorage) Cookies(u *url.URL) string {
    // Retrieve cookies from Redis
    val := r.client.Get(ctx, fmt.Sprintf("cookies:%s", u.Host))
    return val.Val()
}

func (r *RedisStorage) SetCookies(u *url.URL, cookies string) {
    // Store cookies in Redis
    r.client.Set(ctx, fmt.Sprintf("cookies:%s", u.Host), cookies, 0)
}
```

--------------------------------

### Configure Robots.txt Handling

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Choose whether to ignore robots.txt rules or respect them. By default, Colly respects robots.txt.

```go
// Ignore robots.txt (default behavior)
c := colly.NewCollector(colly.IgnoreRobotsTxt())

// Respect robots.txt
c := colly.NewCollector()
// robots.txt is checked by default when IgnoreRobotsTxt=false
```

--------------------------------

### Get Collector Debug Information

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Returns a string representation of the collector, including debug information about request/response counts and registered callbacks.

```go
func (c *Collector) String() string
```

```go
fmt.Println(c.String())
// Output: Requests made: 10 (10 responses) | Callbacks: OnRequest: 1, OnHTML: 2, OnResponse: 1, OnError: 1
```

--------------------------------

### AddURL

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Adds a new URL to the queue to be processed as a GET request. Returns an error if the queue is full and exceeds the storage limit.

```APIDOC
## AddURL

### Description
Adds a new URL to the queue as a GET request.

### Signature
```go
func (q *Queue) AddURL(URL string) error
```

### Parameters
#### Path Parameters
- **URL** (string) - Required - URL to queue

### Returns
- error - Queue is full if storage limit exceeded.

### Example
```go
q.AddURL("https://example.com/page1")
q.AddURL("https://example.com/page2")
```
```

--------------------------------

### Setting Custom Storage in Collector

Source: https://github.com/gocolly/colly/blob/master/_autodocs/storage.md

Demonstrates how to replace the default storage with a custom implementation, such as RedisStorage, by calling the SetStorage method on the collector.

```go
c := colly.NewCollector()

// Create custom storage
redisStorage := &RedisStorage{
    client: redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    }),
}

// Set custom storage
if err := c.SetStorage(redisStorage); err != nil {
    log.Fatal(err)
}
```

--------------------------------

### Check if Colly Queue is Empty

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Returns true if the queue contains no requests. Use this to conditionally start or manage queue processing.

```go
func (q *Queue) IsEmpty() bool
```

```go
if !q.IsEmpty() {
    q.Run(c)
}
```

--------------------------------

### Post-Creation Configuration of Collector Settings

Source: https://github.com/gocolly/colly/blob/master/_autodocs/configuration.md

Demonstrates how to modify Colly collector settings after the collector has been instantiated. This includes changing the user agent, adding headers, adjusting limits, and updating domain restrictions.

```go
c := colly.NewCollector()

// Change user agent
c.UserAgent = "New User-Agent"

// Add headers
c.Headers.Set("X-Custom", "Value")

// Change limits
c.MaxDepth = 5
c.MaxRequests = 1000

// Add new domain restrictions
c.AllowedDomains = []string{"example.com"}
```

--------------------------------

### Enable HTTP Tracing

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Enable HTTP tracing to measure connection and response times. Access trace information within the `OnResponse` callback.

```go
c := colly.NewCollector(colly.TraceHTTP())

c.OnResponse(func(r *colly.Response) {
    if r.Trace != nil {
        fmt.Printf("Connect: %v\n", r.Trace.ConnectDuration)
        fmt.Printf("First byte: %v\n", r.Trace.FirstByteDuration)
    }
})
```

--------------------------------

### Configure Allowed Domains

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Sets a domain whitelist. Only URLs from these domains will be visited. Accepts multiple domains.

```go
c := colly.NewCollector(colly.AllowedDomains("example.com", "test.com"))
```

--------------------------------

### ErrRetryBodyUnseekable Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Illustrates handling ErrRetryBodyUnseekable when attempting to retry a request with a non-seekable request body. Ensure the request body is seekable if retries are expected.

```go
var ErrRetryBodyUnseekable = errors.New("Retry Body Unseekable")
```

```go
req, _ := r.New("POST", url, someReader) // someReader not seekable
err := req.Retry()
// err == colly.ErrRetryBodyUnseekable
```

--------------------------------

### ErrAbortedBeforeRequest Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Returned when Request.Abort() is called within the OnRequest callback, preventing the HTTP request from being sent. Use this to conditionally skip requests before they are made.

```go
var ErrAbortedBeforeRequest = errors.New("Aborted before Do Request")
```

```go
c.OnRequest(func(r *colly.Request) {
    if strings.Contains(r.URL.String(), "block") {
        r.Abort()
    }
})
// Request aborted -> ErrAbortedBeforeRequest
```

--------------------------------

### Create New Queue with Default Storage

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Initializes a new request queue with a specified number of threads and the default in-memory storage.

```go
q, _ := queue.New(2, nil) // 2 threads, default in-memory storage
```

--------------------------------

### ErrAbortedAfterHeaders Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Indicates that a request was aborted in the OnResponseHeaders callback after headers were received but before the body was downloaded. This is useful for skipping requests based on response headers.

```go
var ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
```

```go
c.OnResponseHeaders(func(r *colly.Response) {
    if r.StatusCode == 404 {
        r.Request.Abort() // Skip 404s
    }
})
// Request aborted -> ErrAbortedAfterHeaders
```

--------------------------------

### NewContext

Source: https://github.com/gocolly/colly/blob/master/_autodocs/context.md

Initializes a new Context instance. Contexts are automatically created for each request but can also be created manually.

```APIDOC
## NewContext

### Description
Initializes a new Context instance. Contexts are automatically created for each request but can also be created manually.

### Returns
* Context - New empty context.

### Example
```go
ctx := colly.NewContext()
r.Ctx = ctx
```
```

--------------------------------

### ErrNoURLFiltersMatch Example

Source: https://github.com/gocolly/colly/blob/master/_autodocs/errors.md

Occurs when URLFilters are set and the target URL does not match any of the specified allowed patterns. Ensure your URLFilters correctly include all desired URLs.

```go
var ErrNoURLFiltersMatch = errors.New("No URLFilters match")
```

```go
c := colly.NewCollector(
    colly.URLFilters(regexp.MustCompile("/api/")),
)
err := c.Visit("https://example.com/page")
// err == colly.ErrNoURLFiltersMatch (doesn't match /api/)
```

--------------------------------

### Execute Custom Request with Do

Source: https://github.com/gocolly/colly/blob/master/_autodocs/request.md

Use `Do` to submit a request that has been manually constructed with specific parameters. Ensure all necessary headers and body content are set before calling this method.

```go
req, _ := r.New("POST", "https://example.com", nil)
req.Headers.Set("X-Custom", "Header")
req.Do()
```

--------------------------------

### Configure Performance Settings

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Optimize collector performance by enabling asynchronous requests, limiting body size, setting request timeouts, and defining parallelism rules for domains.

```go
c := colly.NewCollector(
    colly.Async(),                              // Parallel
    colly.AllowURLRevisit(),                    // Skip duplicate checks
    colly.MaxBodySize(1024 * 1024),             // Limit size
)
c.SetRequestTimeout(10 * time.Second)
c.Limit(&colly.LimitRule{
    DomainGlob: "*",
    Parallelism: 10,
})
```

--------------------------------

### Clone Collector Configuration

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Create a new collector instance with the same configuration as an existing one. This is useful for reusing settings while defining different callbacks.

```go
c1 := colly.NewCollector()
c2 := c1.Clone() // Same config, different callbacks
```

--------------------------------

### Get Collector Statistics in Go

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Call the String() method on a collector instance to retrieve statistics about its operation, such as the number of requests made and callbacks triggered.

```go
fmt.Println(c.String())
```

--------------------------------

### Get Colly Queue Size

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Returns the current number of requests in the queue. Useful for monitoring queue load. Handles potential storage errors.

```go
func (q *Queue) Size() (int, error)
```

```go
size, _ := q.Size()
log.Printf("Queue has %d items\n", size)
```

--------------------------------

### Configure Disallowed Domains

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Sets a domain blacklist. URLs from these domains will be skipped. Accepts multiple domains.

```go
c := colly.NewCollector(colly.DisallowedDomains("example.com", "test.com"))
```

--------------------------------

### Configure URL Filters

Source: https://github.com/gocolly/colly/blob/master/_autodocs/quick-start.md

Use DisallowedURLFilters to block specific URLs and URLFilters to only allow URLs matching a pattern. Requires regexp import.

```go
c := colly.NewCollector(
    colly.DisallowedURLFilters(regexp.MustCompile("logout")),
    colly.URLFilters(regexp.MustCompile("/api/")),
)
```

--------------------------------

### Stop Colly Queue Gracefully

Source: https://github.com/gocolly/colly/blob/master/_autodocs/queue.md

Stops the running queue gracefully. Can be called from callbacks to halt processing. Ensure the queue is started before calling stop.

```go
func (q *Queue) Stop()
```

```go
q.Run(c) // in background or separate goroutine

// Later, stop the queue
q.Stop()
```

--------------------------------

### Configure URL Filters

Source: https://github.com/gocolly/colly/blob/master/_autodocs/collector.md

Sets regular expressions that URLs must match to be visited. URLs not matching these will be skipped.

```go
c := colly.NewCollector(colly.URLFilters(regexp.MustCompile("^https://example.com/keep$")))
```