### Install grobotstxt Package (Developers) Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Use this command to get the package if you are not using Go modules. ```bash go get github.com/jimsmart/grobotstxt ``` -------------------------------- ### Install grobotstxt Binary (Webmasters) Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Build and install the standalone 'icanhasrobot' binary for webmasters to test robots.txt rules. ```bash go install github.com/jimsmart/grobotstxt/... ``` -------------------------------- ### RobotsMatcher HandleRobotsStart Method Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked at the start of parsing a robots.txt file. It resets all instance member variables. ```go func (m *RobotsMatcher) HandleRobotsStart() ``` -------------------------------- ### Example Sitemap Extraction Output Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Illustrates the expected output format when extracting sitemaps. ```text [http://example.net/sitemap.xml http://example.net/sitemap2.xml] ``` -------------------------------- ### Use icanhasrobot Tool Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Example of using the 'icanhasrobot' command-line tool to check if a bot is allowed to access a URL based on a robots.txt file. ```bash $ icanhasrobot ~/local/path/to/robots.txt YourBot https://example.com/url user-agent 'YourBot' with URI 'https://example.com/url': ALLOWED ``` -------------------------------- ### Example robots.txt rules for longest match Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Illustrates conflicting Allow and Disallow rules in robots.txt, demonstrating the longest-match strategy where the more specific Disallow rule takes precedence. ```text Allow: / Disallow: /cgi-bin ``` -------------------------------- ### Create New RobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Initializes a new RobotsMatcher instance. ```go func NewRobotsMatcher() *RobotsMatcher ``` -------------------------------- ### Create New Parser Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Initializes a new Parser with the given robots.txt content and a ParseHandler. ```go func NewParser(robotsBody string, handler ParseHandler) *Parser ``` -------------------------------- ### Import grobotstxt Package (Developers) Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Import the package into your Go project to use its functionalities. ```go import "github.com/jimsmart/grobotstxt" ``` -------------------------------- ### HandleRobotsStart Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked at the beginning of parsing a robots.txt file, responsible for resetting all instance member variables. ```APIDOC ## HandleRobotsStart ### Description HandleRobotsStart is called at the start of parsing a robots.txt file, and resets all instance member variables. ### Method func (m *RobotsMatcher) HandleRobotsStart() ``` -------------------------------- ### RobotsMatcher HandleSitemap Method Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked for every 'Sitemap:' line in robots.txt. For RobotsMatcher, this method performs no action. ```go func (m *RobotsMatcher) HandleSitemap(lineNum int, value string) ``` -------------------------------- ### Parse robots.txt with Parser Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Initiates the parsing process for the robots.txt content associated with the Parser. ```go func (p *Parser) Parse() ``` -------------------------------- ### Run Test Coverage Report Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Execute tests and generate an HTML coverage report to analyze test coverage. ```bash go test -coverprofile=coverage.out && go tool cover -html=coverage.out ``` -------------------------------- ### Use icanhasrobot with Multiple User-Agents Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Demonstrates how to pass multiple comma-separated user-agent names to the 'icanhasrobot' tool. ```bash $ icanhasrobot ~/local/path/to/robots.txt Googlebot,Googlebot-image https://example.com/url user-agent 'Googlebot,Googlebot-image' with URI 'https://example.com/url': ALLOWED ``` -------------------------------- ### HandleSitemap Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked for each 'Sitemap:' line in robots.txt. For RobotsMatcher, this action is a no-op. ```APIDOC ## HandleSitemap ### Description HandleSitemap is called for every "Sitemap:" line in robots.txt. For RobotsMatcher, this does nothing. ### Method func (m *RobotsMatcher) HandleSitemap(lineNum int, value string) ``` -------------------------------- ### Match Path Against Robots.txt Pattern Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Implements robots.txt pattern matching. The pattern is anchored to the beginning of the path, and '$' is only special at the end. Ensures acceptable worst-case performance. ```go func Matches(path, pattern string) bool ``` -------------------------------- ### Define MatchStrategy interface Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Defines the MatchStrategy interface, which specifies methods for matching individual lines in a robots.txt file. Each method should return a match priority indicating the degree of match. ```go type MatchStrategy interface { MatchAllow(path, pattern string) int MatchDisallow(path, pattern string) int } ``` -------------------------------- ### HandleUserAgent Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked for each 'User-Agent:' line encountered in the robots.txt file. ```APIDOC ## HandleUserAgent ### Description HandleUserAgent is called for every "User-Agent:" line in robots.txt. ### Method func (m *RobotsMatcher) HandleUserAgent(lineNum int, userAgent string) ``` -------------------------------- ### HandleUnknownAction Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked for any unrecognized lines in robots.txt. For RobotsMatcher, this action is a no-op. ```APIDOC ## HandleUnknownAction ### Description HandleUnknownAction is called for every unrecognized line in robots.txt. For RobotsMatcher, this does nothing. ### Method func (m *RobotsMatcher) HandleUnknownAction(lineNum int, action, value string) ``` -------------------------------- ### Implement MatchDisallow for LongestMatchStrategy Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Implements the MatchDisallow method for the LongestMatchStrategy. ```go func (s LongestMatchStrategy) MatchDisallow(path, pattern string) int ``` -------------------------------- ### Implement MatchAllow for LongestMatchStrategy Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Implements the MatchAllow method for the LongestMatchStrategy. ```go func (s LongestMatchStrategy) MatchAllow(path, pattern string) int ``` -------------------------------- ### NewParser Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Creates a new Parser with the given robots.txt body and handler. ```APIDOC ## NewParser ### Description NewParser creates a new Parser with the given robots.txt body and handler. ### Signature ```go func NewParser(robotsBody string, handler ParseHandler) *Parser ``` ``` -------------------------------- ### Check Agent Allowance with RobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Uses a RobotsMatcher to determine if a user agent is allowed to fetch a specific URI based on the robots.txt content. ```go func (m *RobotsMatcher) AgentAllowed(robotsBody string, userAgent string, uri string) bool ``` -------------------------------- ### HandleAllow Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked for each 'Allow:' line encountered in the robots.txt file. ```APIDOC ## HandleAllow ### Description HandleAllow is called for every "Allow:" line in robots.txt. ### Method func (m *RobotsMatcher) HandleAllow(lineNum int, value string) ``` -------------------------------- ### RobotsMatcher MatchingLine Method Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Returns the line number that matched the rules, or 0 if no rules matched. ```go func (m *RobotsMatcher) MatchingLine() int ``` -------------------------------- ### NewRobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Creates a RobotsMatcher with the default longest-match strategy. ```APIDOC ## NewRobotsMatcher ### Description NewRobotsMatcher creates a RobotsMatcher with the default matching strategy. The default matching strategy is longest-match as opposed to the former internet draft that provisioned first-match strategy. Analysis shows that longest-match, while more restrictive for crawlers, is what webmasters assume when writing directives. For example, in case of conflicting matches (both Allow and Disallow), the longest match is the one the user wants. For example, in case of a robots.txt file that has the following rules: ``` Allow: / Disallow: /cgi-bin ``` it's pretty obvious what the webmaster wants: they want to allow crawl of every URI except /cgi-bin. However, according to the expired internet standard, crawlers should be allowed to crawl everything with such a rule. ### Signature ```go func NewRobotsMatcher() *RobotsMatcher ``` ``` -------------------------------- ### Check Multiple Agent Allowances with RobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Uses a RobotsMatcher to check if any user agent from a list is allowed to fetch a specific URI. ```go func (m *RobotsMatcher) AgentsAllowed(robotsBody string, userAgents []string, uri string) bool ``` -------------------------------- ### Handle Unknown Action in RobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback method for the RobotsMatcher to handle any unrecognized directives found during parsing. ```go func (m *RobotsMatcher) HandleUnknownAction(lineNum int, action, value string) ``` -------------------------------- ### HandleRobotsEnd Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked at the conclusion of parsing the robots.txt file. For RobotsMatcher, this action is a no-op. ```APIDOC ## HandleRobotsEnd ### Description HandleRobotsEnd is called at the end of parsing the robots.txt file. For RobotsMatcher, this does nothing. ### Method func (m *RobotsMatcher) HandleRobotsEnd() ``` -------------------------------- ### Extract Sitemaps from robots.txt Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Parses the robots.txt content and returns a slice of all declared sitemap URLs. ```go func Sitemaps(robotsBody string) []string ``` -------------------------------- ### Parse Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Parses the robots.txt body using a provided handler. ```APIDOC ## Parse ### Description Parse uses the given robots.txt body and ParseHandler to create a Parser, and calls its Parse method. ### Signature ```go func Parse(robotsBody string, handler ParseHandler) ``` ``` -------------------------------- ### RobotsMatcher HandleRobotsEnd Method Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked at the end of parsing the robots.txt file. For RobotsMatcher, this method performs no action. ```go func (m *RobotsMatcher) HandleRobotsEnd() ``` -------------------------------- ### Handle User-Agent Directive in RobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback method for the RobotsMatcher to handle a 'User-agent' directive found during parsing. ```go func (m *RobotsMatcher) HandleUserAgent(lineNum int, userAgent string) ``` -------------------------------- ### HandleDisallow Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback function invoked for each 'Disallow:' line encountered in the robots.txt file. ```APIDOC ## HandleDisallow ### Description HandleDisallow is called for every "Disallow:" line in robots.txt. ### Method func (m *RobotsMatcher) HandleDisallow(lineNum int, value string) ``` -------------------------------- ### Parse robots.txt Content with Handler Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Parses the provided robots.txt content using a custom ParseHandler to process directives. ```go func Parse(robotsBody string, handler ParseHandler) ``` -------------------------------- ### Parser.Parse Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Parses the robots.txt body and emits parse callbacks. ```APIDOC ## Parser.Parse ### Description Parse body of this Parser's robots.txt and emit parse callbacks. This will accept typical typos found in robots.txt, such as 'disalow'. Note, this function will accept all kind of input but will skip everything that does not look like a robots directive. ### Signature ```go func (p *Parser) Parse() ``` ``` -------------------------------- ### Matches Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Matches implements robots.txt pattern matching. Returns true if URI path matches the specified pattern. Pattern is anchored at the beginning of path. '$' is special only at the end of pattern. Since both path and pattern are externally determined (by the webmaster), we make sure to have acceptable worst-case performance. ```APIDOC ## Matches ### Description Matches implements robots.txt pattern matching. Returns true if URI path matches the specified pattern. Pattern is anchored at the beginning of path. '$' is special only at the end of pattern. Since both path and pattern are externally determined (by the webmaster), we make sure to have acceptable worst-case performance. ### Function Signature ```go func Matches(path, pattern string) bool ``` ``` -------------------------------- ### Sitemaps Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Extracts all "Sitemap:" values from the robots.txt content. ```APIDOC ## Sitemaps ### Description Sitemaps extracts all "Sitemap:" values from the given robots.txt content. ### Signature ```go func Sitemaps(robotsBody string) []string ``` ### Example ``` [http://example.net/sitemap.xml http://example.net/sitemap2.xml] ``` ``` -------------------------------- ### Handle Allow Directive in RobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback method for the RobotsMatcher to handle an 'Allow' directive found during parsing. ```go func (m *RobotsMatcher) HandleAllow(lineNum int, value string) ``` -------------------------------- ### Define Parser struct Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Defines the Parser struct, which is used for parsing robots.txt content. ```go type Parser struct { // contains filtered or unexported fields } ``` -------------------------------- ### Check if Agent is Allowed to Fetch URI Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Parses robots.txt content to determine if a specific user agent is permitted to fetch a given URI. Returns false for invalid URIs. ```go func AgentAllowed(robotsBody string, userAgent string, uri string) bool ``` -------------------------------- ### Check if Any Agent in List is Allowed to Fetch URI Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Parses robots.txt content to check if any user agent from a list is allowed to fetch a given URI. Returns false for invalid URIs. ```go func AgentsAllowed(robotsBody string, userAgents []string, uri string) bool ``` -------------------------------- ### Define RobotsMatcher struct Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Defines the RobotsMatcher struct, which uses a MatchStrategy to match robots.txt rules against URIs. It defaults to the longest-match strategy, which is the official Google crawler method. ```go type RobotsMatcher struct { MatchStrategy MatchStrategy // contains filtered or unexported fields } ``` -------------------------------- ### Handle Disallow Directive in RobotsMatcher Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Callback method for the RobotsMatcher to handle a 'Disallow' directive found during parsing. ```go func (m *RobotsMatcher) HandleDisallow(lineNum int, value string) ``` -------------------------------- ### AllowFrequentTypos Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt AllowFrequentTypos enables the parsing of common typos in robots.txt, such as DISALOW. ```APIDOC ## AllowFrequentTypos ### Description AllowFrequentTypos enables the parsing of common typos in robots.txt, such as DISALOW. ### Variable Declaration ```go var AllowFrequentTypos = true ``` ``` -------------------------------- ### RobotsMatcher DisallowedIgnoreGlobal Method Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Returns true if crawling a matching URI is disallowed, ignoring rules for the default user agent and considering only specified user agents. ```go func (m *RobotsMatcher) DisallowedIgnoreGlobal() bool ``` -------------------------------- ### Check Agent Permissions with AgentAllowed Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Use the AgentAllowed function to determine if a specific bot is permitted to access a given URI according to the robots.txt content. ```go import "github.com/jimsmart/grobotstxt" // Contents of robots.txt file. robotsTxt := ` # robots.txt with restricted area User-agent: * Disallow: /members/* Sitemap: http://example.net/sitemap.xml ` // Target URI. uri := "http://example.net/members/index.html" // Is bot allowed to visit this page? ok := grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", uri) ``` -------------------------------- ### RobotsMatcher.AgentAllowed Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Checks if a given URI is allowed for a specific user agent based on robots.txt content. ```APIDOC ## RobotsMatcher.AgentAllowed ### Description AgentAllowed parses the given robots.txt content, matching it against the given userAgent and URI, and returns true if the given URI is allowed to be fetched by the given user agent. AgentAllowed will also return false if the given URI is invalid (cannot successfully be parsed by url.Parse). ### Signature ```go func (m *RobotsMatcher) AgentAllowed(robotsBody string, userAgent string, uri string) bool ``` ``` -------------------------------- ### Enable Frequent Typos in robots.txt Parsing Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Set this variable to true to allow parsing of common typos in robots.txt files, such as 'DISALOW'. ```go var AllowFrequentTypos = true ``` -------------------------------- ### Extract Sitemap URIs with Sitemaps Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt The Sitemaps function extracts all Sitemap URIs listed within a given robots.txt content. ```go sitemaps := grobotstxt.Sitemaps(robotsTxt) ``` -------------------------------- ### RobotsMatcher EverSeenSpecificAgent Method Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Returns true if the robots file explicitly referred to one of the specified user agents when AgentsAllowed() was called. ```go func (m *RobotsMatcher) EverSeenSpecificAgent() bool ``` -------------------------------- ### AgentsAllowed Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt AgentsAllowed parses the given robots.txt content, matching it against the given userAgents and URI, and returns true if the given URI is allowed to be fetched by any user agent in the list. AgentsAllowed will also return false if the given URI is invalid (cannot successfully be parsed by url.Parse). ```APIDOC ## AgentsAllowed ### Description AgentsAllowed parses the given robots.txt content, matching it against the given userAgents and URI, and returns true if the given URI is allowed to be fetched by any user agent in the list. AgentsAllowed will also return false if the given URI is invalid (cannot successfully be parsed by url.Parse). ### Function Signature ```go func AgentsAllowed(robotsBody string, userAgents []string, uri string) bool ``` ``` -------------------------------- ### MatchingLine Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Retrieves the line number that matched the criteria, or 0 if no match was found. ```APIDOC ## MatchingLine ### Description MatchingLine returns the line that matched or 0 if none matched. ### Method func (m *RobotsMatcher) MatchingLine() int ``` -------------------------------- ### RobotsMatcher.AgentsAllowed Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Checks if a given URI is allowed for any of the specified user agents. ```APIDOC ## RobotsMatcher.AgentsAllowed ### Description AgentsAllowed parses the given robots.txt content, matching it against the given userAgents and URI, and returns true if the given URI is allowed to be fetched by any user agent in the list. AgentsAllowed will also return false if the given URI is invalid (cannot successfully be parsed by url.Parse). ### Signature ```go func (m *RobotsMatcher) AgentsAllowed(robotsBody string, userAgents []string, uri string) bool ``` ``` -------------------------------- ### Define LongestMatchStrategy struct Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Defines the LongestMatchStrategy struct, which implements the default robots.txt matching strategy by returning the maximum number of characters matched by a pattern as its match priority. ```go type LongestMatchStrategy struct{} ``` -------------------------------- ### Define ParseHandler interface Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Defines the ParseHandler interface, which provides callbacks for directives found during robots.txt parsing. These callbacks are invoked by the Parse() function in the order they appear in the file. ```go type ParseHandler interface { HandleRobotsStart() HandleRobotsEnd() HandleUserAgent(lineNum int, value string) HandleAllow(lineNum int, value string) HandleDisallow(lineNum int, value string) HandleSitemap(lineNum int, value string) HandleUnknownAction(lineNum int, action, value string) } ``` -------------------------------- ### AgentAllowed Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt AgentAllowed parses the given robots.txt content, matching it against the given userAgent and URI, and returns true if the given URI is allowed to be fetched by the given user agent. AgentAllowed will also return false if the given URI is invalid (cannot successfully be parsed by url.Parse). ```APIDOC ## AgentAllowed ### Description AgentAllowed parses the given robots.txt content, matching it against the given userAgent and URI, and returns true if the given URI is allowed to be fetched by the given user agent. AgentAllowed will also return false if the given URI is invalid (cannot successfully be parsed by url.Parse). ### Function Signature ```go func AgentAllowed(robotsBody string, userAgent string, uri string) bool ``` ### Example ```go // Example usage would go here, but is not provided in the source. // Output: false ``` ``` -------------------------------- ### DisallowedIgnoreGlobal Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Checks if the current URI is disallowed, ignoring global user agent rules and considering only specified user agents. ```APIDOC ## DisallowedIgnoreGlobal ### Description Returns true if we are disallowed from crawling a matching URI. Ignores any rules specified for the default user agent, and bases its results only on the specified user agents. ### Method func (*RobotsMatcher) DisallowedIgnoreGlobal() bool ``` -------------------------------- ### RobotsMatcher.Disallowed Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Returns true if crawling a matching URI is disallowed. ```APIDOC ## RobotsMatcher.Disallowed ### Description Disallowed returns true if we are disallowed from crawling a matching URI. ### Signature ```go func (m *RobotsMatcher) Disallowed() bool ``` ``` -------------------------------- ### Check if URI is Disallowed Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Returns true if the current matching URI is disallowed from crawling. ```go func (m *RobotsMatcher) Disallowed() bool ``` -------------------------------- ### EverSeenSpecificAgent Source: https://pkg.go.dev/github.com/jimsmart/grobotstxt Determines if the robots file explicitly referred to one of the specified user agents during the AgentsAllowed() call. ```APIDOC ## EverSeenSpecificAgent ### Description Returns true iff, when AgentsAllowed() was called, the robots file referred explicitly to one of the specified user agents. ### Method func (*RobotsMatcher) EverSeenSpecificAgent() bool ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.