### Install grobotstxt for Developers

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Use this command to get the grobotstxt package if you are not using Go modules.

```bash
go get github.com/jimsmart/grobotstxt
```

--------------------------------

### Install grobotstxt Standalone Binary

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Build and install the 'icanhasrobot' binary for webmasters to test robots.txt rules.

```bash
go install github.com/jimsmart/grobotstxt/...
```

--------------------------------

### Install and Use icanhasrobot CLI Tool

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

The icanhasrobot binary allows webmasters to test local robots.txt files against user-agents and URLs from the shell. It exits with codes 0 (allowed), 1 (disallowed), or 2 (bad input).

```bash
# Install the tool
go install github.com/jimsmart/grobotstxt/...

# Test a single user-agent
$ icanhasrobot /path/to/robots.txt Googlebot https://example.com/public/page
user-agent 'Googlebot' with URI 'https://example.com/public/page': ALLOWED

# Test a restricted path
$ icanhasrobot /path/to/robots.txt BadBot https://example.com/members/profile
user-agent 'BadBot' with URI 'https://example.com/members/profile': DISALLOWED

# Test multiple user-agents (comma-separated, no spaces)
$ icanhasrobot /path/to/robots.txt Googlebot,Googlebot-Image https://example.com/images/photo.jpg
user-agent 'Googlebot,Googlebot-Image' with URI 'https://example.com/images/photo.jpg': ALLOWED

# Show help
$ icanhasrobot --help

# Use exit code in a shell script
icanhasrobot robots.txt MyCrawler https://example.com/target && echo "Crawling..." || echo "Blocked."
```

--------------------------------

### Low-Level Path Pattern Matching with Matches

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

The Matches function tests if a URI path conforms to a single robots.txt pattern. Patterns are anchored at the start, '*' matches any sequence of characters, and '$' anchors to the end. This function implements Google's core matching algorithm.

```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	// Wildcard mid-path
	fmt.Println(grobotstxt.Matches("/files/report_2024.pdf", "/files/*.pdf"))
	// Output: true

	// End-of-string anchor
	fmt.Println(grobotstxt.Matches("/index.html", "/index.html$"))
	// Output: true

	fmt.Println(grobotstxt.Matches("/index.html?q=foo", "/index.html$"))
	// Output: false

	// Prefix match (no anchor)
	fmt.Println(grobotstxt.Matches("/admin/settings", "/admin/"))
	// Output: true

	// No match
	fmt.Println(grobotstxt.Matches("/public/page", "/private/"))
	// Output: false
}
```

--------------------------------

### Matches - Low-level path pattern matching

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Tests whether a URI path matches a single robots.txt pattern string. The pattern is anchored at the start of the path; '*' is a wildcard matching any sequence of characters, and '$' at the end anchors the pattern to the end of the path.

```APIDOC
## Matches

### Description
Tests if a given URI path matches a robots.txt pattern.

### Parameters
- **path** (string) - The URI path to test.
- **pattern** (string) - The robots.txt pattern to match against.

### Returns
- **bool** - True if the path matches the pattern, false otherwise.

### Pattern Syntax
- `*`: Wildcard matching any sequence of characters.
- `$`: Anchors the pattern to the end of the path.
```

--------------------------------

### Run Go Tests with Coverage

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Execute tests and generate an HTML coverage report to analyze test coverage.

```bash
go test -coverprofile=coverage.out && go tool cover -html=coverage.out
```

--------------------------------

### Import grobotstxt Package in Go

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Import the grobotstxt package into your Go project to use its functionalities.

```go
import "github.com/jimsmart/grobotstxt"
```

--------------------------------

### Instantiate RobotsMatcher for Repeated Use

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Use NewRobotsMatcher() to create a reusable matcher instance for checking multiple URIs against a robots.txt file. This provides access to lower-level metadata like the matched line number and whether a specific agent was seen. Note: this struct is not concurrency-safe.

```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	robotsTxt := `
User-agent: FooBot
Disallow: /secret/

User-agent: *
Allow: /
`

matcher := grobotstxt.NewRobotsMatcher()

	// Check multiple URIs with the same matcher instance
	uris := []string{
		"https://example.com/secret/data",
		"https://example.com/public/page",
		"https://example.com/secret/other",
	}

	for _, uri := range uris {
		allowed := matcher.AgentAllowed(robotsTxt, "FooBot", uri)
		fmt.Printf("% -45s -> allowed=%v, matchLine=%d, seenSpecific=%v\n",
			uri, allowed, matcher.MatchingLine(), matcher.EverSeenSpecificAgent())
	}
}
```

--------------------------------

### icanhasrobot (CLI Tool)

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

A standalone command-line executable for testing robots.txt rules. It allows webmasters to check if a specific user-agent can access a given URL based on a local robots.txt file.

```APIDOC
## icanhasrobot (CLI Tool)

### Description
A command-line utility that tests a single URL and user-agent against a robots.txt file. Its inputs and outputs are compatible with Google's original tool.

### Usage
`icanhasrobot <path/to/robots.txt> <user-agent> <url>`

### Parameters
- **`<path/to/robots.txt>`** (string) - The local file path to the robots.txt file.
- **`<user-agent>`** (string) - The user-agent string to test. Can be a comma-separated list of user agents.
- **`<url>`** (string) - The URL to check access for.

### Example Usage
```bash
$ icanhasrobot ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with URI 'https://example.com/url': ALLOWED
```

### Example with Multiple User Agents
```bash
$ icanhasrobot ~/local/path/to/robots.txt Googlebot,Googlebot-image https://example.com/url
user-agent 'Googlebot,Googlebot-image' with URI 'https://example.com/url': ALLOWED
```

### Installation
```bash
go install github.com/jimsmart/grobotstxt/...
```

By default, the binary will be installed to `$GOPATH/bin` or `$GOBIN`.
```

--------------------------------

### Custom Robots.txt Parsing with Parse and ParseHandler

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Implement the ParseHandler interface to create custom logic for processing robots.txt directives. The Parse function tokenizes the robots.txt content and calls the appropriate handler methods for each directive encountered, allowing for detailed inspection and collection of rules.

```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

// auditHandler collects all directives for inspection.
type auditHandler struct {
	userAgents []string
	allows     []string
	disallows  []string
	sitemaps   []string
	unknown    []string
}

func (h *auditHandler) HandleRobotsStart()                              { *h = auditHandler{} }
func (h *auditHandler) HandleRobotsEnd()                                {}
func (h *auditHandler) HandleUserAgent(_ int, v string)                 { h.userAgents = append(h.userAgents, v) }
func (h *auditHandler) HandleAllow(_ int, v string)                     { h.allows = append(h.allows, v) }
func (h *auditHandler) HandleDisallow(_ int, v string)                  { h.disallows = append(h.disallows, v) }
func (h *auditHandler) HandleSitemap(_ int, v string)                   { h.sitemaps = append(h.sitemaps, v) }
func (h *auditHandler) HandleUnknownAction(_ int, action, value string) { h.unknown = append(h.unknown, action+":"+value) }

func main() {
	robotsTxt := `
# Example robots.txt
User-agent: *
Disallow: /cgi-bin/
Allow: /cgi-bin/public/
Crawl-delay: 10

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml
`

	h := &auditHandler{}
	grobotstxt.Parse(robotsTxt, h)

	fmt.Println("User-Agents:", h.userAgents)
	// Output: User-Agents: [* Googlebot]

	fmt.Println("Disallows:  ", h.disallows)
	// Output: Disallows:   [/cgi-bin/]

	fmt.Println("Allows:     ", h.allows)
	// Output: Allows:      [/cgi-bin/public/ /]

	fmt.Println("Sitemaps:   ", h.sitemaps)
	// Output: Sitemaps:    [https://example.com/sitemap.xml]

	fmt.Println("Unknown:    ", h.unknown)
	// Output: Unknown:     [Crawl-delay:10]
}
```

--------------------------------

### RobotsMatcher - Reusable stateful matcher

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Instantiate RobotsMatcher directly with NewRobotsMatcher() for repeated use or when access to lower-level result metadata is needed. Note: it is not concurrency-safe.

```APIDOC
## NewRobotsMatcher

### Description
Creates a new instance of RobotsMatcher.

### Usage
```go
m := grobotstxt.NewRobotsMatcher()
```

## RobotsMatcher.AgentAllowed

### Description
Checks if a given agent is allowed to access a URI based on the provided robots.txt content.

### Parameters
- **robotsTxt** (string) - The content of the robots.txt file.
- **agent** (string) - The user-agent string to check.
- **uri** (string) - The URI to check.

### Returns
- **bool** - True if the agent is allowed, false otherwise.

## RobotsMatcher.MatchingLine

### Description
Returns the line number in the robots.txt file that determined the matching result.

### Returns
- **int** - The line number of the match.

## RobotsMatcher.EverSeenSpecificAgent

### Description
Indicates whether the specific agent was ever seen in the robots.txt file during the last check.

### Returns
- **bool** - True if the specific agent was seen, false otherwise.
```

--------------------------------

### Extract Sitemap URIs from robots.txt

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Use the `Sitemaps` function to parse robots.txt content and retrieve a slice of all `Sitemap:` URIs. It returns an empty slice if no Sitemap directives are found. Ensure the robots.txt content is correctly formatted.

```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	robotsTxt := `
User-agent: *
Disallow: /tmp/

Sitemap: https://example.com/sitemap-index.xml
Sitemap: https://example.com/news-sitemap.xml
Sitemap: https://cdn.example.com/images/sitemap.xml
`

	sitemaps := grobotstxt.Sitemaps(robotsTxt)
	fmt.Println(sitemaps)
	// Output: [https://example.com/sitemap-index.xml https://example.com/news-sitemap.xml https://cdn.example.com/images/sitemap.xml]

	fmt.Println(len(sitemaps))
	// Output: 3

	// robots.txt with no Sitemap directives
	empty := grobotstxt.Sitemaps("User-agent: *\nDisallow: /\n")
	fmt.Println(empty)
	// Output: []
}
```

--------------------------------

### Extract Sitemap URIs with grobotstxt

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Use the Sitemaps function to extract all Sitemap URIs listed in a robots.txt file.

```go
sitemaps := grobotstxt.Sitemaps(robotsTxt)
```

--------------------------------

### Test URL Access with icanhasrobot

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Use the 'icanhasrobot' binary to check if a specific user-agent is allowed to access a URL based on a robots.txt file.

```bash
$ icanhasrobot ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with URI 'https://example.com/url': ALLOWED
```

```bash
$ icanhasrobot ~/local/path/to/robots.txt Googlebot,Googlebot-image https://example.com/url
user-agent 'Googlebot,Googlebot-image' with URI 'https://example.com/url': ALLOWED
```

--------------------------------

### Check URL Access for a Single User Agent

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Use `AgentAllowed` to parse robots.txt content and determine if a specific user agent is permitted to fetch a given URI. It handles URI normalization and returns false for invalid URIs. Ensure the robots.txt content and URI are correctly formatted.

```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	robotsTxt := `
User-agent: *
Disallow: /private/
Disallow: /members/*

User-agent: Googlebot
Allow: /members/public/

Sitemap: https://example.com/sitemap.xml
`

	// Disallowed for all bots
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "https://example.com/private/data.html"))
	// Output: false

	// Allowed by the wildcard rule
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "https://example.com/public/page.html"))
	// Output: true

	// Googlebot has an explicit Allow override for /members/public/
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "Googlebot", "https://example.com/members/public/profile.html"))
	// Output: true

	// FooBot has no override, so the wildcard Disallow applies
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "https://example.com/members/index.html"))
	// Output: false

	// Invalid URI returns false
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "://not a valid uri"))
	// Output: false
}
```

--------------------------------

### Parse / ParseHandler - Custom streaming parser

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Tokenizes a robots.txt body and calls the corresponding method on the supplied ParseHandler for every directive encountered. Implement the ParseHandler interface to build custom processing logic.

```APIDOC
## Parse

### Description
Parses the robots.txt content and calls methods on the provided handler for each directive.

### Parameters
- **robotsTxt** (string) - The content of the robots.txt file.
- **handler** (ParseHandler) - An implementation of the ParseHandler interface.

### ParseHandler Interface
- `HandleRobotsStart()`: Called at the beginning of parsing.
- `HandleRobotsEnd()`: Called at the end of parsing.
- `HandleUserAgent(line int, value string)`: Called for each `User-agent` directive.
- `HandleAllow(line int, value string)`: Called for each `Allow` directive.
- `HandleDisallow(line int, value string)`: Called for each `Disallow` directive.
- `HandleSitemap(line int, value string)`: Called for each `Sitemap` directive.
- `HandleUnknownAction(line int, action string, value string)`: Called for any unknown directives.
```

--------------------------------

### Toggle Typo Tolerance with AllowFrequentTypos

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Controls whether common misspellings of robots.txt directives are accepted. Set to false for strict RFC compliance.

```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	// robots.txt with a common typo: "Disalow" instead of "Disallow"
	robotsTxt := `
User-agent: *
Disalow: /typo-protected/
`

	uri := "https://example.com/typo-protected/secret"

	// Default: typos are tolerated — the misspelled directive is honoured
	grobotstxt.AllowFrequentTypos = true
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot", uri))
	// Output: false

	// Strict mode: typo is ignored — access is allowed
	grobotstxt.AllowFrequentTypos = false
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot", uri))
	// Output: true

	// Restore default
	grobotstxt.AllowFrequentTypos = true
}
```

--------------------------------

### Check Agent Allowance with grobotstxt

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Use the AgentAllowed function to determine if a user-agent is permitted to access a given URI according to the provided robots.txt content.

```go
import "github.com/jimsmart/grobotstxt"

// Contents of robots.txt file.
robotsTxt := "\n    # robots.txt with restricted area\n\n    User-agent: *\n    Disallow: /members/*\n\n    Sitemap: http://example.net/sitemap.xml\n"

// Target URI.
uri := "http://example.net/members/index.html"


// Is bot allowed to visit this page?
ok := grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", uri)
```

--------------------------------

### Check URL Access for Multiple User Agents

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Utilize `AgentsAllowed` to check if any user agent from a provided list is permitted to fetch a given URI. This is useful for crawlers that present multiple user-agent tokens. Ensure the robots.txt content and the list of agents are correctly formatted.

```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	robotsTxt := `
User-agent: Googlebot-Image
Allow: /images/

User-agent: *
Disallow: /images/
Disallow: /admin/
`

	agents := []string{"Googlebot", "Googlebot-Image"}

	// Googlebot-Image is explicitly allowed, so the combined check is true
	fmt.Println(grobotstxt.AgentsAllowed(robotsTxt, agents, "https://example.com/images/photo.jpg"))
	// Output: true

	// Neither agent has access to /admin/
	fmt.Println(grobotstxt.AgentsAllowed(robotsTxt, agents, "https://example.com/admin/panel"))
	// Output: false

	// No rules for /public/, so access is allowed by default
	fmt.Println(grobotstxt.AgentsAllowed(robotsTxt, agents, "https://example.com/public/page.html"))
	// Output: true
}
```

--------------------------------

### Sitemaps

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Extracts all Sitemap URIs declared within a robots.txt file. This is a helper function to easily retrieve sitemap locations.

```APIDOC
## Sitemaps

### Description
Parses the provided robots.txt content and extracts all declared Sitemap URIs.

### Function Signature
`func Sitemaps(robotsTxtContent string) []string`

### Parameters
- **robotsTxtContent** (string) - The content of the robots.txt file.

### Returns
- **[]string** - A slice of strings, where each string is a URL of a declared sitemap.

### Example Code
```go
import "github.com/jimsmart/grobotstxt"

robotsTxt := `
    Sitemap: http://example.net/sitemap.xml
    Sitemap: http://example.com/sitemap_index.xml
`

sitemaps := grobotstxt.Sitemaps(robotsTxt)
// sitemaps will be ["http://example.net/sitemap.xml", "http://example.com/sitemap_index.xml"]
```
```

--------------------------------

### AgentAllowed

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Checks if a single user agent is allowed to fetch a given URL based on the provided robots.txt content. It handles URL normalization and returns false for invalid URIs.

```APIDOC
## AgentAllowed 

### Description
Parses the given robots.txt content and returns `true` if the specified user agent is permitted to fetch the given URI. The URI may be a standard Go UTF-8 string; normalisation to RFC 3986 encoding is handled internally. Returns `false` for invalid URIs.

### Method
`AgentAllowed(robotsTxt string, userAgent string, uri string) bool`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	robotsTxt := `
User-agent: *
Disallow: /private/
Disallow: /members/*

User-agent: Googlebot
Allow: /members/public/

Sitemap: https://example.com/sitemap.xml
`

	// Disallowed for all bots
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "https://example.com/private/data.html"))
	// Output: false

	// Allowed by the wildcard rule
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "https://example.com/public/page.html"))
	// Output: true

	// Googlebot has an explicit Allow override for /members/public/
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "Googlebot", "https://example.com/members/public/profile.html"))
	// Output: true

	// FooBot has no override, so the wildcard Disallow applies
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "https://example.com/members/index.html"))
	// Output: false

	// Invalid URI returns false
	fmt.Println(grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "://not a valid uri"))
	// Output: false
}
```

### Response
#### Success Response (bool)
- `true`: The user agent is allowed to fetch the URL.
- `false`: The user agent is not allowed to fetch the URL or the URI is invalid.

#### Response Example
`true` or `false`
```

--------------------------------

### AgentsAllowed

Source: https://context7.com/jimsmart/grobotstxt/llms.txt

Checks if any user agent from a provided list is allowed to fetch a given URL. This is useful for crawlers that present multiple user-agent tokens.

```APIDOC
## AgentsAllowed 

### Description
Parses the given robots.txt content and returns `true` if **any** of the supplied user agents is permitted to fetch the given URI. Useful when a single crawler presents multiple user-agent tokens.

### Method
`AgentsAllowed(robotsTxt string, userAgents []string, uri string) bool`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```go
package main

import (
	"fmt"
	"github.com/jimsmart/grobotstxt"
)

func main() {
	robotsTxt := `
User-agent: Googlebot-Image
Allow: /images/

User-agent: *
Disallow: /images/
Disallow: /admin/
`

	agents := []string{"Googlebot", "Googlebot-Image"}

	// Googlebot-Image is explicitly allowed, so the combined check is true
	fmt.Println(grobotstxt.AgentsAllowed(robotsTxt, agents, "https://example.com/images/photo.jpg"))
	// Output: true

	// Neither agent has access to /admin/
	fmt.Println(grobotstxt.AgentsAllowed(robotsTxt, agents, "https://example.com/admin/panel"))
	// Output: false

	// No rules for /public/, so access is allowed by default
	fmt.Println(grobotstxt.AgentsAllowed(robotsTxt, agents, "https://example.com/public/page.html"))
	// Output: true
}
```

### Response
#### Success Response (bool)
- `true`: At least one of the provided user agents is allowed to fetch the URL.
- `false`: None of the provided user agents are allowed to fetch the URL.

#### Response Example
`true` or `false`
```

--------------------------------

### AgentAllowed

Source: https://github.com/jimsmart/grobotstxt/blob/main/README.md

Checks if a given user agent is allowed to access a specific URI based on the provided robots.txt content. This function is a direct port of the core functionality from Google's robots.txt parser.

```APIDOC
## AgentAllowed

### Description
Determines if a specified user agent is permitted to access a given URI according to the rules in a robots.txt file.

### Function Signature
`func AgentAllowed(robotsTxtContent string, userAgent string, uri string) bool`

### Parameters
- **robotsTxtContent** (string) - The content of the robots.txt file.
- **userAgent** (string) - The user agent string to check.
- **uri** (string) - The URI to check access for.

### Returns
- **bool** - `true` if the user agent is allowed, `false` otherwise.

### Example Code
```go
import "github.com/jimsmart/grobotstxt"

robotsTxt := `
    User-agent: *
    Disallow: /members/*
`
uri := "http://example.net/members/index.html"

ok := grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", uri)
// ok will be false
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.